Archived - Impact Evaluation of the ecoENERGY for Aboriginal and Northern Communities

Date: November 2010
Project Number: 1570-7/09065

PDF Version (329 Kb, 64 Pages)

Table of Contents

• List of Acronyms
• Definition of Commonly Used Terms
• Executive Summary
• Management Response / Action Plan
• 1. Introduction
• 2. Evaluation Methodology
• 3. Evaluation Findings - Relevance
• 4. Evaluation Findings - Performance (Effectiveness / Success)
• 5. Evaluation Findings - Performance (Lessons Learned and Best Practices)
• 6. Evaluation Findings - Performance (Economy and Efficiency)
• 7. Evaluation Findings - Performance (Alternatives)
• 8. Conclusions and Recommendations
• Appendix A - EANC Logic Model
• Appendix B - Evaluation Matrix
• Appendix C - EANC Performance Measurement Strategy
• Appendix D - Terms of Reference

List of Acronyms

Definition of Commonly Used Terms

Aboriginal and Northern Community Action Program (ANCAP). A predecessor to ecoENERGY, ANCAP provided funding to recipients to undertake climate change adaptation planning projects, infrastructure vulnerability assessments, risk assessments, renewable and non-renewable resource sector engagement and raising awareness of impacts of climate change, and energy use and production among Aboriginal and northern communities.

Commissioned. Renewable energy and energy efficiency projects are considered commissioned once they go through a series of tests following their construction. Once commissioned, the projects are ready to be fully operational.

Community energy plans. These plans are created at the municipal or community level to address energy needs. Many community energy plans incorporate sustainable energy and GHG reduction strategies.

Criteria Air Contaminants (CACs). Are a group of pollutants specified by Environment Canada as airborne pollutants that cause smog and acid rain. In 2002, EC added the gases to the National Pollutant Release Inventory (NPRI) to promote cleaner air outcomes in Canada. CACs are primarily generated through the burning of fossil fuels (EC 2006; EC, 2009b). These pollutants adversely affect human health and the environment. CACs generally refer to sulphur oxides, nitrogen oxides, particulate matter, volatile organic compounds, carbon monoxide, ammonia, ground-level ozone and secondary particulate matter (EC, 2006).

Fossil fuel. Fossil fuels are those derived from coal, petroleum and natural gas. The burning of fossil fuels generates greenhouse gasses, namely carbon dioxide. Fossil fuels are a non-renewable energy source.

Greenhouse Gas (GHGs) Emissions. Greenhouse gases are those gaseous constituents of the atmosphere, both natural and anthropogenic, that absorb and emit radiation at specific wavelengths within the spectrum of infrared radiation emitted by the Earth's surface, the atmosphere and clouds. This property causes the greenhouse effect. Water vapour (H2O), carbon dioxide (CO2), nitrous oxide (N2O), methane (CH4) and ozone (O3) are the primary greenhouse gases in the earth's atmosphere. Moreover, there are a number of entirely human-made greenhouse gases in the atmosphere, such as sulphur hexafluoride (SF6), hydrofluorocarbons (HFCs), and perfluorocarbons (PFCs).

Megawatt. Megawatt is a unit of power equal to one million watts.

"Off-grid" energy production. This term refers to self-sufficiency in energy production without reliance on a public utility energy distribution grid.

Renewable / sustainable energy sources. Unlike fossil fuels, these energy sources are continuously replenished through natural means. Several examples include hydro facilities, wind turbines, solar electric systems, and biomass plants.

RETScreen. This clean energy analysis software permits users to evaluate energy production and savings, costs, emission reductions, financial viability and risk for various types of Renewable-energy and Energy-efficient Technologies (RETs). The software is provided free of charge.

Tonnes/Megatonnes. This is a measurement of GHGs, namely carbon dioxide in the atmosphere. One megatonne is equal to one million tonnes.

Executive Summary

The ecoENERGY for Aboriginal and Northern Communities Program (EANC) evaluation is expected to provide conclusions regarding the relevance and performance of the program. The Evaluation, Performance Measurement and Review Branch (part of the Audit and Evaluation Sector) initiated this evaluation in May 2010. The Branch contracted the services of PRA Inc. to provide assistance during all stages of the evaluation process. The results of this evaluation will be rolled up to the Clean Energy Theme (CET) level evaluation. Natural Resources Canada (NRCan) is the CET lead and as such is responsible for the completion of the thematic level evaluation in fiscal year 2010-2011. The EANC evaluation will also support renewal of contribution authorities in 2010–11.

EANC is a four-year, $15-million Grants and Contributions initiative that began on April 1, 2007. The program provides support for renewable energy, energy efficiency, and community energy plan projects in Aboriginal and northern communities. As of March 31, 2010, EANC has funded 76 projects, including 55 renewable energy projects, eight energy efficiency projects, and 13 community energy plans, for a total spending of $11.2 million.

Key findings / conclusions from the evaluation are as follows:

Relevance

The EANC is a relevant federal and INAC program with the goal of reducing Greenhouse Gas (GHG) and Criteria Air Contaminants (CAC) emissions. The EANC contributes to the achievement of government priorities (the Clean Air Agenda) by supporting the use of renewable energy and energy efficient technologies. It is aligned with both the federal government clean energy priorities and Indian and Northern Affairs Canada's (INAC) strategic objectives relating to Aboriginal "economic well being and prosperity". It also complements activities of other stakeholders (e.g. provinces/territories).

Due to its responsibilities and its experience with Aboriginal and northern communities, INAC, through the EANC has a clear role in clean energy projects as it pertains to Aboriginals. While the provinces/territories have jurisdiction over energy generation, funding for EANC projects is relevant due to the fact that there is limited funding available for these types of projects in Aboriginal and northern communities. Further, there is an ongoing need for the EANC as it is leading to results in social, environmental, and economic development benefits for the communities that participate in the program.

Performance

Effectiveness (i.e., Success)

At the outset, the EANC had four planned activities:

1. Communications, networking, and outreach activities—targeted to Departmental Capital Program Staff in the regions and external stakeholders;
2. Procedural support for the acceptance, review, and approval of projects;
3. Technological and financial advice on the application of renewable energy and energy efficiency technology in northern and Aboriginal communities; and
4. Technical review, assessment, and approval of projects.

The evaluation found that most of the EANC's planned activities have been implemented at some level. At its beginning, the program developed monitoring, reporting, and project selection processes as well as implementing performance measures. The information collected by the program do not allow for a thorough assessment of some outcomes. For example, all of the performance measures included indicators and almost all were measured quantitatively. However, many of these performance measures did not have targets, hence, measuring the extent to which they were successful proved difficult; this was also compounded by the fact that the program developed too many indicators, and did not provide the necessary information at the right level.

Another important issue is that the program did not focus on a key objective of the program: doing or measuring the reduction of CACs emissions. As a result, no information related to main CACs outcomes was collected. This shows the need for better articulated priorities as well as how the program is designed together in order to achieve its goal.

In reference to outputs, the quantitative and qualitative data available showed that EANC was delivering on its intended outputs and leading towards the achievement of its expected outcomes. For example, in those cases where targets were available, the finding shows that EANC was achieving its outputs and outcomes, with the exception of expected support for community energy planned projects, where it fell short. Collaboration between utilities and Aboriginal and northern communities and the development of those partnerships, while not identified as expected outcomes at the start of the EANC, became one of the main benefits of the program.

A key feature of the EANC was its ability to provide early funding in order to get projects started promptly. This allowed, for each dollar invested by the EANC, the leveraging of $26 from other sources, and thus, enabled the successful completion of EANC funded projects. However, the results have been mixed because some projects were funded at 100 percent by INAC while others were only able to progress to the implementation stage as a result of limited funding (i.e. inability to leverage the total funds).

Lessons Learned and Best Practices

Key informants and program recipients interviewed provided suggestions related to lessons learned and best practices. These suggestions revolved around the importance of dedicating regional staff to the program, having ongoing relationships with INAC staff, and developing community energy plans before embarking on large renewable energy and energy efficiency projects.

Economy and Efficiency

The economy aspect of the EANC was not addressed as it required financial information for each output of projects and operating costs in order to estimate the economy of resources used. These data were not collected by the program and not available elsewhere.

The program's efficiency and cost-effectiveness were difficult to determine. While comparisons can be made with similar programs, often the information from other programs is not available. However, because the programs are inherently different from each other, making comparisons between them is difficult.

Although, many key informants were unsure of whether the EANC was cost-effective at reducing GHGs and CACs, program recipients interviewed believed that the program is cost‑effective as it gets good return on its small investments in multi-million dollar projects.

Alternatives

In the absence of any specific alternative that is compatible with aboriginal community capacity, the main alternatives, noted in the literature and identified in the case studies, involve close cooperation and joint funding of projects and initiatives with other federal departments and other orders of government. Some key informants believed that decentralizing the program delivery to the regional level would be beneficial for the program.

It is recommended that INAC

1. Move towards better integration of renewable energy and energy efficiency considerations with other jurisdictions and with INAC's programs, such as the Community Economic Opportunities Program, Major Resource and Economic Development, Strategic Partnership Initiative within economic development, with community infrastructure, and with other related community development initiatives.
2. Clarify the overall direction of the program, determining whether it is intended to operate as a program to provide incentives for the launch of projects in Aboriginal and northern communities (i.e. setting the groundwork) or to contribute to the achievement of eco-energy objectives.
   1. Refocus the performance measurement strategy to strengthen the capacity to collect the appropriate data on the results of both the projects funded and the program.
   2. Align administration and reporting requirements to the amount invested and the level of risk (program and recipient).

Management Response / Action Plan

Project Title: Evaluation of the ecoENERGY for Aboriginal and Northern Communities Program
Project #: 1570-7/09065

The Management Response and Action Plan for the Evaluation of the ecoENERGY for Aboriginal and Northern Communities Program were approved by the Evaluation, Performance Measurement and Review Committee on November 18, 2010.

1. Introduction

1.1 Overview

This is the final report of an impact evaluation of the ecoENERGY for Aboriginal and Northern Communities (EANC) Program. The program offers financial support to Aboriginal and northern communities seeking to invest in "clean" energy projects, including renewable energy generation, energy efficiency, and community energy plans. The Evaluation, Performance Measurement and Review Branch (EPMRB), part of the Audit and Evaluation Sector, Indian and Northern Affairs Canada (INAC), initiated this evaluation in May 2010. EPMRB contracted the services of PRA Inc. to provide assistance during all stages of the evaluation process.

The EANC is part of the Clean Energy Theme (CET) of the Clean Air Agenda (CAA). INAC is responsible for evaluating the EANC. The results of this evaluation will be rolled up to the CET level evaluation. Natural Resources Canada is the CET lead and as such is responsible for the completion of the thematic level evaluation in fiscal year 2010-2011. The EANC evaluation will also support renewal of contribution authorities in 2010-11.

The evaluation was expected to provide evidence based conclusions regarding relevance, design and delivery, results/success, and cost-effectiveness/alternatives of INAC's approach towards providing assistance to Aboriginal and Northerners (Aboriginal) as it pertains to the EANC.

This report has eight sections. This introduction describes the program and evaluation process. Section 2 presents the methodology associated with the study and includes a description of the scope and timing of the evaluation, a summary of the evaluation issues and questions addressed in this report, along with a description of the various methods used to collect evaluation data and findings. It also discusses the limitations to the methodology, mitigation strategies, and quality assurance used to support this study. Sections 3, 4, 5, 6, and 7 include the most critical information relating to the evaluation of the EANC, as they summarize the findings that have emerged during the data collection process. Section 3 specifically explores the relevance of the EANC, while sections 4, 5, 6, and 7 focus on the performance (results/success, lessons learned and best practices, efficiency, and economy and alternatives) of the program. Finally, Section 8 provides conclusions and recommendations.

1.2 Program Profile

1.2.1 Background and Description

The EANC is one of eight programs under the CET in the CAA. [Note 1] The CAA is the Government of Canada's action plan to achieve progress and sustainability for Canada's environment. Initiated in April 2007, the CAA is a four year, $1.9-billion interdepartmental initiative that engages multiple environmental issues to secure a clean and healthy natural environment for all Canadians. [Note 2] The departments that are a part of the CAA are supported by the Horizontal Management, Accountability and Reporting Framework, "which defines the governance structure and roles and responsibilities of departments and the various interdepartmental committees, and identifies key operational issues for horizontal implementation." [Note 3]

The ecoENERGY initiatives under the CET are "a set of measures to promote smarter energy use, increase clean energy supply, and support the development and deployment of clean energy technologies." [Note 4] These have been designed to ease the transition to lower emissions reductions that will be required under the regulatory aspects of the CAA in the long term. [Note 5]

The EANC is a four year, $15-million Grants and Contributions initiative that began on April 1, 2007. [Note 6] The program targets Aboriginal and northern communities and includes over 700 qualifying communities, comprising 140 "off-grid" (diesel/gas/natural gas-dependent electricity generation) communities [Note 7] and approximately 700,000 Canadians. [Note 8] The program funds clean energy projects in Aboriginal and northern communities that fall under the following three categories:

• Integrating energy-efficient or renewable energy technologies into new or existing community infrastructure, including improving diesel generation efficiency in off-grid communities; installing waste heat recovery systems; and installing district heating applications, solar thermal systems, solar hot water systems, and improved energy and heating systems. A maximum of $100,000 is available per project.
• Developing large renewable energy projects, which include small and micro hydro facilities, wind turbines, solar electric systems, and biomass plants. These projects are usually past the feasibility stage and may include a range of partners in the funding. These projects are expected to have greenhouse gas (GHG) emissions reductions well above 4,000 tonnes over the project life cycle of approximately 20 years. [Note 9] Up to $250,000 is available for each project.
• Community energy baselines and community energy plans, which will provide technical and financial support to eligible organizations to develop feasibility studies and action plans that address clean energy in their community. A maximum of $5,000 to $15,000 is available per community. [Note 10], [Note 11]

Other than the maximum dollar amounts, the EANC also has a maximum percentage of project funding. The program will fund no more than 30 percent of the project development costs and capital costs in on-grid communities while off-grid communities may receive up to a maximum of 50 percent of project costs. [Note 12] It is expected that the program will assist in the development of over 200 megawatts (MW) of installed electrical generation capacity, which is projected to lead to a reduction of 1.3 megatonnes (Mt) of GHGs through the displacement of natural gas, coal, and diesel-electric generation. [Note 13]

The EANC was modelled after the Aboriginal and Northern Community Action Program (ANCAP), a similar program that was delivered by INAC between 2003 and 2007. ANCAP promoted clean energy projects by providing support systems, capacity building, and awareness in Aboriginal and northern communities, and engaging stakeholder partnerships. ANCAP resulted in the delivery of over 100 projects. [Note 14]

At the outset, it was expected that the EANC would fund 22 to 35 projects per year, including 6 to 15 renewable energy and energy efficiency projects and 16 to 20 community energy plan projects. [Note 15] As of March 31, 2010, 76 projects, within 64 Aboriginal and northern communities, had been funded under the EANC for an annual average of just over 25 projects per year and a total funding amount of $7.6 million (see table 2 , on section 1.2.3). The projects funded included 55 renewable energy projects, eight energy efficiency projects, and 13 community energy plan projects. [Note 16] Ofthose projects, six renewable energy and four energy efficiency projects had been commissioned as of March 31, 2010, and it is expected that another eight renewable energy and five energy efficiency projects will be commissioned by the end of the program on March 31, 2011. [Note 17]

1.2.2 Program Objectives and Expected Outcomes

In June 2007, a Results-based Management and Accountability Framework and Risk Based Audit Framework (RMAF/RBAF) was developed for the EANC. In this framework, the EANC had two objectives and these were to facilitate renewable energy and energy efficiency projects that will:

• Reduce GHG emissions by a projected 1.3 Mt over the life cycle of projects supported by the initiative [Note 18], and
• Decrease the emissions of Criteria Air Contaminants (CACs), resulting in improved air quality. [Note 19]

These objectives are expected to be achieved through funding of projects that reduce or displace natural gas, coal and diesel generation of electricity, through more energy efficient technologies and increased use of renewable energy. The EANC is also expected to result in other social, environmental, and economic development benefits for the communities that participate in the program. [Note 20]

Project funding terms are to be developed for each category of project. A part of the terms is that EANC funding can be allocated to one or more of the following project activities directly related to a project:

• Community climate change capacity development;
• GHG emissions estimation;
• GHG emissions reductions, monitoring and verification, or emissions credit trading;
• Technology application of energy efficiency and renewable energy technologies;
• Human resource development related to energy efficiency and renewable technologies;
• Project management components related to GHG estimation and management; and
• Pre-feasibility/feasibility Studies, environmental assessments, and resource analyses. [Note 21]

The following EANC activities were also identified as essential to achieving the objectives:

• Communications, networking, and outreach activities; targeted to Departmental Capital Program Staff in the regions to help identify and develop viable proposals, and external stakeholders who will be important partners in project support and delivery;
• Procedural support for the acceptance, review, and approval of projects;
• Technological and financial advice on the application of renewable energy and energy efficiency technology in northern and Aboriginal communities; and
• Technical review, assessment, and approval of projects. [Note 22]

Expected Outcomes

The implementation of the activities noted above is expected to contribute to the achievement of a series of immediate, intermediate and long-term outcomes. The following are the EANC's planned expected outcomes:

Immediate outcomes

• Aboriginal and northern communities are engaged and participating in the ecoENERGY program.
• Improved Aboriginal and northern technical and management skills for clean energy.
• Successfully implemented renewable energy and energy efficiency projects.

Intermediate outcomes

• Aboriginal and northern communities are equipped with knowledge and tools to increase energy efficiency of community infrastructure.
• Decrease in emissions of GHGs and CACs in northern and Aboriginal communities.
• Increased energy efficiency and use of renewable energy in northern and Aboriginal communities.
• More reliable energy infrastructure in northern and Aboriginal communities.
• Energy cost savings in northern and Aboriginal communities.

Long-term outcomes

• Reduced GHG emissions and CACs in Aboriginal and northern communities (1.3 megatonnes of GHG reductions over the life cycle of projects supported by the initiative).
• Sustainable and healthy northern and Aboriginal communities. [Note 23]

The logic model (Appendix A) outlines how the activities align with the expected outputs and outcomes of the EANC. Section 3 (Evaluation Findings) discusses these activities and the results achieved.

1.2.3 Program Management

The project selection process for the EANC includes the key steps to decide on funding. The Project Review Committee (i.e., Large Energy Project Working Group (LEPWG) is responsible for reviewing proposed projects, ensuring recommended projects are feasible and meet the program's eligibility criteria, and providing project approval recommendations to the director. The LEPWG, comprised of representatives from Economic Development, Capital Programs, Northern Affairs sectors in the Department and a technical contractor (i.e., third party), [Note 24] meets several times a year.

1.2.4 Program Resources

The program Grants and Contributions funding is drawn from two authorities:

1. Contribution Authority 334 - Contribution for promoting the safe use, development, conservation and protection of the North's natural resources, ($8 million), which is associated with Strategic Outcome – The People (Healthy Northern Communities); and
2. Contribution Authority 377 - Payments to support Indians, Inuit and Innu for the purpose of supplying public services in areas such as: capital facilities and maintenance, $2 million, associated with Strategic Outcome – The People (Community Infrastructure).

At the outset of the EANC, the resources were expected to be allocated as outlined in Table 1.

FTEs = Full-time Equivalents; O&M = Operation and Management; EBP = Employee Benefit Plan; Gs&C = Grants and Contributions; PWGSC = Public Works and Government Services Canada Accommodation Costs.
Source: INAC. (2007). EcoEnergy for Aboriginal and Northern Communities Initiative Results-based Management and Accountability Framework and Risk Based Audit Framework.

Actual spending of the program up to 2009-10 appears in Table 2.

O&M = Operation and Management; EBP = Employee Benefit Plan; Gs&C = Grants and Contributions; PWGSC = Public Works and Government Services Canada Accommodation Costs.
Source: INAC. (2009). HMARF Reporting Template 2008-09. Provided by program May 12, 2010; INAC. (2009). Program: ecoENERGY for Aboriginal and Northern Communities. Provided by program May 12, 2010; INAC Climate Change Division. (2010). 2007/2008 Annual Report. Provided by program August 10, 2010.

Overall program spending per fiscal year has remained close to expected expenditures. While the program spent less than expected in its first year when it was starting, it spent slightly more than expected in the following two fiscal years. The Grants and Contributions of the program have followed a similar pattern to overall spending. The program has $3,782,471 to spend in 2010-11, which is approximately $30,000 more than expected at the outset of the program.

2. Evaluation Methodology

2.1 Evaluation Scope and Timing

The evaluation examined the EANC activities undertaken between April 1, 2007, and March 31, 2010. INAC's Evaluation, Performance Measurement and Review Committee approved Terms of Reference on May 14, 2010, and fieldwork was conducted between May and September of 2010.

2.2 Evaluation Issues and Questions

In line with the Terms of Reference, the evaluation focused on the following issues:

Relevance

• Continued Need
  • A1.1 What is the environmental need the program is trying to address?
• Alignment with Government Priorities
  • A2.1 Is the program consistent with government priorities and INAC strategic objectives?
• Alignment with Federal Roles and Responsibilities
  
  • A3.1 Is there a legitimate, appropriate, and necessary role for INAC in clean energy projects for Aboriginal and northern communities?

Performance

• Effectiveness (i.e., design and delivery, results/success)
  • B1.1 To what extent have EANC activities been implemented as planned?
  • B1.2 To what extent does program management identify and act on lessons learned, best practices, and required improvements in the design and delivery of the program?
  • B1.3 To what extent were realistic performance measures established at the start of the program?
  • B1.4 Is the project selection process effective?
  • B1.5 Are there monitoring and reporting measures in place? If yes, are they appropriate? Do they provide timely and useful information?
  • B1.6 To what extent has the program delivered its required outputs?
  • B1.7 To what extent has the program realized its immediate, intermediate, and long-term outcomes?
  • B1.8 To what extent were program funds leveraged to ensure effective implementation of projects?
  • B1.9 To what extent has the program contributed to constructive interaction and collaboration between the utilities and Aboriginal and northern communities?
  • B1.10 What unintended outcomes (positive or negative) have emerged from the implementation of the program?
  • B1.11 Were there any best practices or lessons learned from the program?
• Demonstrations of Efficiency and Economy (i.e., cost-effectiveness / alternatives)
  • B2.1 Is the program within its budget? If not, what accounts for the variances (positive and negative)?
  • B2.2 Does EANC provide a cost-effective method to reduce GHGs and CACs?
  • B2.3 Compared to EANC, are there alternative approaches that are just as or more efficient and/or effective at reducing GHGs and CACs in Aboriginal and northern communities?

2.3 Evaluation Methodology

The evaluation's findings and conclusions are based on the analysis and triangulation of multiple lines of evidence (see also Appendix B, Evaluation Matrix). This subsection describes these various methods.

2.3.1 Data Sources

Four data sources were used to support the EANC evaluation.

Document and data review:

The document review involved a thorough examination of program files, background documents, agreements, performance measurement materials, and further documentation recovered from the INAC website. The data review included the assessment and analysis of the EANC's financial and performance measurement data.

Literature review:

The literature review utilized information provided by INAC, a Google Scholar  [Note 25] search of the academic and technical literature, a review of key journals in the field, and an examination of studies conducted in other domestic and international jurisdictions. The review cited major and representative studies, as opposed to offering a complete discussion and enumeration of all research in an area.

Key informant interviews:

INAC staff provided PRA Inc. with a list of potential interviewees. Using the list, PRA Inc. completed interviews with 10 representatives from:

• INAC staff (five); and
• External stakeholders (two recipients, one Federal Agency representative, and two consultants who provided guidance to proponents in the development of proposals and managed the project) for a total of five.

PRA Inc. conducted eight of the interviews via telephone in the respondents' preferred language. One other interview was conducted in person with two representatives from INAC staff. Each interview lasted between 45 and 60 minutes. With respondents' permission, interviews were audio-recorded to ensure accuracy in reporting.

Before PRA Inc. contacted the interviewees, INAC's EPMRB informed them, by email, of the evaluation and invited them to participate in an interview. Before the interview, PRA Inc. provided the representatives with a copy of the relevant interview guide to help them prepare their responses. Separate INAC manager and external stakeholder guides were designed to capture representatives' opinions on a variety of topics concerning the EANC programming.

Once the interview notes were typed, they were sent to the interviewee for verification. Interviewees stated either that the notes were fine and accurate or provided additional insights and clarifications.

Throughout the report, the scale provided in Table 3 below will be used to identify the proportion of respondents and the frequency of responses offering a particular perspective and to analyze interview data.

Case studies:

The EANC staff provided PRA Inc. with a list of potential case study projects. Using this list, PRA Inc. completed the eight case studies outlined in Table 4.

The case studies provided a way to collect detailed information on a sub-set of activities funded through the EANC to illustrate activities and results. The case studies were selected using the materiality, location, and type of project representation. Based on those criteria, five renewable energy projects were selected, two energy efficiency and one community energy plan.

PRA Inc. conducted eight case study interviews via telephone, with program recipients in their preferred official language. Each interview lasted between 45 and 60 minutes. With respondents' permission, interviews were audio-recorded to ensure accurate reporting. The conduct of each case study also involved a review of the project file/documentation and related information.

The process used is the same as the one which governed the key informant interview. After completion of the program recipient interviews, information from the proponents was combined with information from the case study project files to develop the case study summary. The program recipients interviewed received a draft case study report for verification and revision, as necessary. [Note 26]

2.3.2 Limitations of the Methodology and Data

1. Quality of project data. Project information, including RETScreen analysis, the nature of the fossil fuels displaced, and the net benefits of the renewable energy and energy efficiency projects, speaks directly to the contribution of the EANC to the CET and CAA. However, this information was not collected for all projects prior to 2009/10.
   
   Mitigation: The evaluation conducted a document review and performed key informant interviews and case studies to support quantitative analysis. The quantitative data collected as part of the EANC's project database is more complete for 2009-2010 than the two previous fiscal years and this information was used to the greatest extent possible.
   
   Impact on evaluation: These gaps in the data prevent a complete quantitative impact analysis of the program, as reporting does not capture the full contribution of the projects. However, the impact is limited, as more than, one line of inquiry has been used, creating a sufficient base of information to assess program results. Data gaps are discussed throughout the findings to clearly show what the limitations in analysis are.
2. Performance data. A major limitation of the study is the lack of data concerning socio-economic and CACs outcomes. Some performance measures cannot be assessed quantitatively. As well, many of the performance measures did not have targets.
   
   Mitigation: Qualitative information was collected in addition to the available quantitative data in order to assess results to the greatest extent possible.
   
   Impact on evaluation:  The main impact of this limitation is that, in some cases, qualitative data was used as a substitute where quantitative data would have been more meaningful. Success in reducing CAC's – a key outcome of the program – was not assessed via the Performance Measurement Strategy and data was unavailable for this evaluation.
3. Number of total interviews with recipients.  Due to the geographical distribution of the recipients (Ontario and British Columbia receiving most of EANC funding), as well as the overall size of the funds allocated to the EANC, relative to the nature of the program, only 10 interviews (eight program recipients and two key informants) were performed out of a total of 64 recipients.
   
   Mitigation: The evaluation triangulated the responses of interviewees with available program and project data gathered through the file and document reviews. Increased reliance on documentary evidence and administrative data strengthened the capacity to respond to all questions and increased the validity of the report. Efforts were taken to ensure that the selected case studies represented the categories of projects funded through the program and interviews targeted well-informed respondents who were able to speak to key issues.
   
   Impact on evaluation: The main risk of this limitation is that the perspectives of recipients were not adequately represented. This, in turn, could lead to a misunderstanding of the true impact of the program at the community level. Nevertheless, many of the central outcomes of the program rely predominantly on quantitative data (e.g.GHG reductions). Sufficient information was gathered to inform findings on outcomes more appropriately addressed through qualitative data (e.g. the development of technical and management skills). In addition, one interview group was not disproportionately relied upon in order to avoid a biased result.
4. Attribution of the results
   4.1 Role of provincial/territorial (P/T) governments and utilities. P/T initiatives (government and utilities) in energy efficiency and energy sustainability is rapidly growing and starting to eclipse federal policy and programming. Most provincial utilities have demand-side programs (incentives to limit energy use) and mechanisms to increase the proportion of renewable energy that may be sold to the utility to reduce energy use. The P/T power utilities, especially those with significant hydro capacity such as British Columbia and the Yukon, are well positioned to make significant contributions to supporting local generation initiatives.
   
   Mitigation: PRA Inc. has collected much information from provinces and utilities through this and previous projects. This context is essential for a complete understanding of the EANC. The scope of this P/T activity is identified in the report, but not quantified in a way that permits detailed analysis.
   
   Impact on evaluation: The fact that there are multiple levels of government providing programming in this area makes it difficult to establish attribution between outcomes and the activities of the EANC and other funders. The issue of duplication is discussed below in the findings section in relation to roles and responsibilities in section A3.1.
   
   4.2 Early stage of the program. Most of the projects have just been completed or are being completed.
   
   Mitigation: Project information provided a good indication of whether results are on track to being achieved. Case studies offered a more in depth and comprehensive perspectives of projects examined.
   
   Impact on evaluation: The impact on the evaluation is limited. Many of the immediate and intermediate outcomes are well-supported by available data. Through modelling and forecasting, there is evidence to support the long-term outcome of GHG reduction.
5. Measurementof GHG emissions.
   5.1 Approach used to measure GHG emissions. A main outcome for this program is to reduce GHG emissions. The reduction in GHGs arising from a clean energy project is rarely measured directly. Rather, these outcomes are estimated based on a series of technical assumptions (modelling or simulations) about how the new project will reduce GHGs. These technical assumptions rest on "laboratory" conditions that may not reflect actual implementation experience.
   
   The projection of how the EANC projects contribute to CET and CAA outcomes reflects this "modelling strategy," as opposed to the direct measurement of GHG reductions. No alternative exists to these "simulated" outcomes, making this an important qualification to the assessment of outcomes of any policy that seeks to mitigate the increase in GHGs.
   
   It is also the case that outcomes reflect projected impacts from the project proposals and not actual measurement after completion. Therefore, any differences that occur during the implementation phase are not captured in the GHG reduction estimations.
   
   Mitigation: A review of how the EANC calculated its GHG reduction estimates was conducted to ensure that it presents current best practice. The only mitigation would be to replace the projected measures with actual measures, using energy audits before and after project initiation. Though this is clear in principle, none of the ecoENERGY suite of programs under the CET has funded such follow-up activity.
   
   Impact on evaluation: The use of projected GHG reductions to measure the outcome follows common practice across the federal government for similar programs. Hence, using this approach permits comparability between programs.

2.4 Roles, Responsibilities, and Quality Assurance

The EPMRB and PRA Inc. worked collaboratively during the design, data collection, and analytical phases of this evaluation study. Each methodology described in Section 2.3 resulted in a technical report. The EPMRB, as the project authority, reviewed each technical report and submitted comments to PRA Inc. before the reports were finalized. The technical reports are used to develop this report.

The report was validated by the program and went through peer review within EPMRB to ensure its quality.

3. Evaluation Findings - Relevance

The EANC is a relevant federal program under CAA. Its focus is on reducing GHG and CACs emissions through reduced use of fossil fuel by implementing renewable sources of electricity generation and reducing energy use through lower consumption as a result of increased efficiency.

INAC's role in clean energy projects, due to its responsibility and experience with Aboriginal and northern communities, is appropriate and the EANC supports INAC's strategic objectives. Although the P/Ts have jurisdiction on energy generation, the EANC contribution is recognized by many recipients as being critical for renewable energy and energy efficiency projects.

A1.1 What is the environmental need the program is trying to address?

The scientific consensus is that recent climate change has been substantially affected by the increased use of fossil fuels, which has led to increases in GHGs and CACs that in turn have contributed to rising temperatures through the greenhouse effect. The remoteness also increases the "carbon footprint" by virtue of the requirement to transport supplies such as diesel to the communities.

The program objective is the reduction of the amount of GHGs and CACs emitted by Aboriginal and northern communities through the installation of renewable energy and energy efficiency technologies and the development of community energy plans. While scientific evidence shows that climate change will likely have a greater adverse impact on Aboriginal and northern communities compared to other communities in Canada, the GHG emissions produced in Aboriginal and northern communities account for no more than about one percent of the national production (as per EC 2005 GHG emissions by province). Therefore, mitigating GHG/CACs emissions for northern and Aboriginal communities can only have a small overall effect on GHG emissions reduction in Canada.

A2.1 Is the program consistent with government priorities and INAC strategic objectives?

The EANC is consistent with the federal government's priorities and INAC's strategic objectives.

Consistency with federal government priorities

The objectives of the EANC are to reduce GHG and CACs emissions. However, the EANC did not collect data on CACs emission. The following points demonstrate how the activities of the EANC are consistent with federal priorities.

• The 2007 and 2008 Speeches from the Throne made commitments to reductions in GHG emissions. Specifically, in the 2008 Speech from the Throne, the federal government committed to reducing Canada's total GHG emissions by 20 percent by 2020. [Note 27] The 2009 Speech from the Throne discusses addressing climate change andpursuing a balanced approach to reducing emissions. Specifically, the Speech indicates that Canada will try to mitigate climate change by "leading the world in clean electricity generation." [Note 28] One of the long-term outcomes and objectives of the EANC is to reduce GHG emissions and one of its main functions is to support the installation of renewable energy projects in Aboriginal and northern communities.
• Supporting the commitments made in the Speeches from the Throne, several federal budgets have earmarked funding allocations aimed at reducing GHG emissions and strengthening economic development in the North and in Aboriginal communities. [Note 29]
• The EANC is part of the Clean Energy theme under the CAA.

Consistency with INAC strategic objectives

As per 2007-2008 INAC Report on Plan and Priorities, renewable energy/energy efficiency objectives are aligned with the Healthy Northern Communities strategic objectives as they address emissions of CACs and GHG that harm human health. As well, the RMAF/RBAF indicates that the program's objectives are consistent with, and contribute to INAC's departmental objectives:

• Providing assistance to Aboriginal and northern communities to invest in physical assets that mitigate health and safety risks;
• Ensuring that activities are undertaken in an environmentally sound and sustainable manner; and
• Equipping Northerners, their governments, and a number of organizations and stakeholders, as well as First Nations and Inuit communities and organizations, with the ability to carry out activities that complement departmental efforts related to safe use, development, conservation, and preservation of the North's natural resources and the implementation of northern contaminants, climate change, and sustainable development strategy initiatives. [Note 30]

A3.1 Is there a legitimate, appropriate, and necessary role for INAC in clean energy projects for Aboriginal and northern communities?

INAC has a unique area of federal mandate and responsibility regarding Aboriginal and northern communities. Many of the key informants agreed that the EANC must be delivered by INAC because of the latter's relationship with Aboriginal and northern communities.

The broad nature of energy issues allows the involvement of P/T Crown energy agencies and the regulated utilities sector as these offer powerful policy instruments. The role of the P/T governments and utilities requires monitoring by the federal government/EANC program to ensure that its programs do not overlap or duplicate the activity of other orders of government.

It is also worth noting that where the federal government is not involved through the EANC, provinces and territories may not step in to promote renewable energy and energy efficiency in Aboriginal communities. Furthermore, due to the costs of renewable energy and energy efficiency projects, key informants indicated that all available funding is critical for advancing the goal of increased use of renewable energy. Almost all of the key informants and about half of the program recipients also noted that the EANC funding is often the initial commitment received by the project. By this standard, many of their projects would likely not have proceeded or would have proceeded on a smaller scale without the EANC funding to initiate public funding. This leveraging by the EANC further reinforces its value and the importance of INAC participation.

4. Evaluation Findings - Performance (Effectiveness / Success)

4.1 Activities

Most of the EANC's planned activities have been implemented at some level. The program developed monitoring, reporting, and project selection processes as well as implementing performance measures at its start. The information collected as part of the performance measurement strategy does not allow a thorough assessment of some outcomes.

Evidence is available that management identifies main improvements and acts on those.

B1.1 To what extent have EANC activities been implemented as planned?

Most of the EANC planned activity has been implemented at some level. At the outset of the program, INAC had the following four planned activities:

1. Communications, networking, and outreach activities—targeted to Departmental Capital Program Staff in the regions and external stakeholders;
2. Procedural support for the acceptance, review, and approval of projects;
3. Technological and financial advice on the application of renewable energy and energy efficiency technology in northern and Aboriginal communities; and
4. Technical review, assessment, and approval of projects.

Program management developed indicators to show how INAC planned to measure whether the activities had been implemented; however, no targets were set. While some quantitative information concerning the implementation of the activities is available, it does not always align directly with the indicators. The remaining evidence is qualitative. Due to these factors, it is difficult to fully measure the "extent" to which each activity has been implemented.

Almost all of the INAC key informants indicated that communication has been good overall at the Headquarters and regional level, while the remaining only mentioned that it was good at the regional level. At the Headquarters level, the key informants noted that a useful brochure had been developed and that staff have attended several events across the country. At the regional level, the key informants indicated that professional material had been developed and that regional staff attended several information sessions and participated in intergovernmental meetings.

All of the INAC key informants and almost all program recipients interviewed said that INAC staff was helpful and quick to respond to questions during the application process. They indicated that the staff is always willing and able to provide support to communities to help them complete an application for a project. The program recipients also confirmed that they enjoyed a good working relationship with INAC staff during the application process.

In 2009–10, INAC received and responded to 88 inquiries concerning the EANC. Some of the INAC key informants, however, noted that experts provided technical and financial advice and conduct general technical reviews when necessary.

Over the first three years of the EANC (2007–08 to 2009–10), INAC received and screened 111 applications for EANC funding. A total of 76 (68.5 percent) of those applications were approved for funding

At the project level, as per the RMAF, the program staff is responsible for monitoring the recipients through a combination of activities. These activities occur according to resources available.

B1.2 To what extent does program management identify and act on lessons learned and best practices?

The "lessons learned and best practices" have been covered in B1.11. In reference to "required improvements in the design and delivery of the program", a management review conducted in 2009 made recommendations. According to the document and file reviews and key informants interviews, management addressed the main improvements listed in the 2009 management review, such as quarterly reports that provide evidence towards the achievement of objectives and key activities of the program, quantitative data collected as part of the EANC's project database is more complete for 2009–10 than the two previous fiscal years, and the Operational Management Guide was finalized/last updated on August 19, 2009.

B1.3 To what extent were realistic performance measures established at the start of the program?

Many of the performance measures were established at the start of the program and indicators were identified and data were being collected by the program (see Appendix C). However, main outcomes such as CACs was not addressed and there are too many indicators, often repetitive and not providing the right level of information.

The responses to evaluation questions B1.1, B1.6, and B1.7 provide evidence that almost all of the expected results were achieved and can be measured quantitatively. Only the activity communications, networking, and outreach activities… and the intermediate outcome more reliable energy infrastructure in northern and Aboriginal communities are not being measured quantitatively by the program.

Many of the INAC key informants were unfamiliar with the performance measures of the program.

B1.4 Is the project selection process effective?

The EANC project selection process seems to be effective and timely. The average approval time for the case study projects was about 1.5 months, which program recipients viewed as acceptable.

All of the case study projects were completed successfully as per file review/completion report and program recipients interviewed. All of the INAC key informants agreed that good criteria had been developed for the selection of projects. However, two of the projects received EANC funding for early stages of their projects and they have been unable to secure funds to begin actual construction. While the existence of other funding (actual or potential) is a criterion for funding that is reviewed as a part of the EANC project selection process, these two projects have been unable to leverage the necessary dollars.

Many INAC key informants also indicated that the program could do more outreach to the communities to increase the number of proposals submitted and to create a strategy to better focus on the overall efforts of the program. However, the data shows that 24 project submissions were received in 2007-2008, 34 in 2008-2009 and 56 in 2009-2010. It represents an increase of 133 percent.

B1.5 Are there monitoring and reporting measures in place? If yes, are they appropriate? Do they provide timely and useful information?

Monitoring and reporting measures for EANC projects were clearly established at the start of the program and according to almost all of the program recipient interviewed and key informants, they are appropriate. However, some of the INAC key informants said that the information being captured from projects was not measuring the true successes of the projects. It must also be noted that the annual reports were completed more than one year after the end of fiscal year whereas the quarterly reports were timely.

Interim project-level activity reports and project closure reports:

Seven of the eight case studies provided interim reports and all of the case studies provided closure reports. Many of the program recipients said the reporting process and requirements for the EANC were straightforward and some other program recipients did not indicate any issues or difficulties with the reporting process.

Almost all of the INAC key informants said that the reporting and monitoring was appropriate and collected all of the necessary information to fulfill the requirements of the Contribution Agreements. However, some of the INAC key informants also mentioned that the Contribution Agreements did not set guidelines for the collection of important information on economic impact, financial impact, actual GHG reduction, and related community outcomes, which show some of the true successes of the projects. In addition, the EANC does not measure actual GHG emission performance after project completion neither is there any reference to a decrease in CACs emissions. This is critical to a proper net impact analysis because estimates made during the application phase may not accurately represent the project's true GHG reductions.

Performance measurement and results measurement:

INAC staff produces an annual report that combines the performance measurement of the EANC and INAC's Climate Change Adaptation Program. Both of these programs are under the CAA. The first annual report (2007–08) was finalized in July 2010. The 2008–09 annual report is not being released yet as it is awaiting final approval.

The annual report includes description of the programs, their objectives, their performances in the relative year, and what is expected in the next year for the programs. As it is covering only the first year of operation, the information regarding results is limited.

Summary reports and status reports:

INAC's program staff develops quarterly reports, done on a timely basis and in table format, which provide the following information for the EANC's objectives and key activities: milestones/performance indicators and targets, results to date and those planned for the next quarter, risks/challenges/mitigation strategies, accountabilities, and status.

INAC must also meet the internal reporting requirements for the EANC, which include providing updates for the annual Report on Plans and Priorities, the annual Departmental Performance Report, the Northern Affairs Organization quarterly report, and the Departmental Sustainable Development Strategy. [Note 31]

4.2 Outputs

Quantitative evidence shows that the EANC has been delivering its outputs. However, since targets were not set for many of the outputs and some of the data collection only began recently, it is difficult to measure the extent to which some of the outcomes have been achieved.

Targets were set for the number of renewable energy, energy efficiency, and community energy plan projects the program planned to fund per year. The program met and exceeded its expected funding of renewable energy and energy efficiency projects per year, but fell short of its community energy plan expectations.

B1.6 To what extent has the program delivered its required outputs?

Most of the required outputs were delivered.

Communication products and promotional materials

A website communicating the EANC to staff and external stakeholders has been developed and is being maintained; however, key informants' opinion of the quality and usefulness of the website differs. While some of the key informants indicated that the website is useful and the information is easily accessible, some of the other key informants indicated that the website was difficult to find and not that useful.

In total, up to 763 brochures were handed out at events attended between September 2008 and present. [Note 32] Almost all key informants confirmed that they had seen promotional materials, and many of them specifically noted the EANC brochure. A few key informants also mentioned seeing the Sharing Knowledge for a Better Future document, which publishes EANC success stories. They noted that this document was very useful for external stakeholders. Only a few key informants indicated that they had not seen any EANC promotional materials other than what was available on the website.

Established 'clean energy network'

As of September 2008, INAC program staff began keeping detailed information concerning the clean energy events they could and did attend. Three in 2007–08, two in 2008–09, and 13 in 2009–10, in total, in 2008–09 and 2009–10, INAC program staff attended 15 out of a possible 48 events. Events attended by the staff in those two fiscal years included conferences (n=6), partnerships (n=1), fora (n=4), workshops (n=1), and meetings (n=3), which included organizations such as the Arctic Energy Alliance, P/T governments, the Canadian Institute, Ontario Waterpower Association, the Atlantic Policy Congress of First Nation Chiefs Secretariat, and many others. [Note 33]

Almost all key informants agreed that they had seen INAC program staff at clean energy events. The attendance of EANC program staff at clean energy events is a good way to advertise the program to external stakeholders. The 763 brochures distributed by the program staff occurred at these events. However, it is not possible to assess if "clean energy network" was/were established and the effectiveness of it.

The horizontal report on this initiative provides information on results. Although it is not clear how these results were measured, during the 2007–08 period the following were indicated:

• Regional and external contacts identified and engaged as part of the 'clean energy network'; and
• Over 200 stakeholders made aware of opportunities available through the program. [Note 34]

Advice on technical feasibility and project management skills to proponents

The only data available is for 2007–08; INAC program staff reviewed and gave advice to 20 Aboriginal and northern community projects. While these quantitative measures indicate EANC activities, they do not indicate what kind of advice was provided. Almost all key informants agreed that the program did not provide proponents with advice on technical feasibility and project management skills. Some of the key informants noted that this was the case because INAC does not have the technical background necessary to advise proponents in this area.

Renewable energy projects are funded in accordance with program objectives

In total, the 55 renewable energy projects accounted for 84 percent of the EANC's Grants and Contributions dollars in the first three years of the program.

Almost all key informants agreed that the program has been successful at funding renewable energy projects to date.

Energy efficiency projects are funded in accordance with program objectives

In total, the eight energy efficiency projects accounted for 13 percent of the EANC's Grants and Contributions dollars in the first three years of the program. The program met or exceeded its goal regarding the number of renewable energy and energy efficiency projects funded each year.

Many key informants indicated that they were aware of energy efficiency projects funded under the EANC. Several of the respondents noted, however, that they would like to see more of these projects in the communities.

Community energy plans are funded in accordance with program objectives

In total, the 13 community energy plans accounted for three percent of the EANC's Grants and Contributions dollars in the first three years of the program. The low percentage of funding is not surprising, since the maximum allotment for a community energy plan is $15,000, while for energy efficiency and renewable energy projects it is up to a maximum of $100,000 or $250,000.

Many key informants indicated awareness of community energy plans funded under the EANC. Some key informants indicated that they did not see any community energy plans funded under the EANC in their region.

The program fell short of its community energy plan expectations by funding fewer than 20 projects in any of its first three years.

4.3 Immediate, Intermediate, and Long-Term Outcomes

Many of the projected outcomes of the EANC are expected to be achieved. Significantly, even though collaboration between the utilities sector and Aboriginal and northern communities, and the development of those partnerships, were not part of the expected outcomes outlined at the start of the EANC, it became one of the main benefits of the program.

A key feature of the EANC is its willingness to provide early funding to get projects started. For each EANC dollar, it leveraged $26 from other sources. However, the case studies show evidence of successful and unsuccessful leveraging of funds to implement projects.

B1.7 To what extent has the program realized its immediate, intermediate, and long-term outcomes?

The quantitative data shows, and many key informants agree, that the immediate, intermediate, and long-term outcomes of the program are being or will be achieved. Some of the outcomes do not have indicated targets and, therefore, measuring the extent to which the outcomes have been achieved is difficult. The target of the long-term outcomes, expected reduction of GHG emissions of 1.3 Mt over the life cycle of the projects, is expected to be met by the projects funded under the EANC.

At the outset of the program, INAC had outlined three immediate, five intermediate, and two long-term outcomes.

Immediate outcomes

Many of the immediate outcomes have been achieved.

Aboriginal and northern communities are engaged and participating in the ecoENERGY program

Over the first three years of the program, INAC received 111 applications for EANC funding. The number of applications received by INAC increased each year, with 24 applications in 2007–08, 34 applications in 2008–09, and 53 applications in 2009–10. The continued increase in applications for the EANC shows an upward trend in the awareness of the program in Aboriginal and northern communities. INAC is expecting another 50 applications in 2010–11.

More generally, almost all key informants and all program recipients interviewed agreed that they have witnessed Aboriginal and northern communities engaging and participating in the program in their regions. Some key informants cautioned, however, that this has just started to happen as a result of the EANC program. Few key INAC informants noted that they would like to see more consistency in Aboriginal and northern community engagement nationally, instead of just in specific regions.

Improved Aboriginal and northern technical and management skills for clean energy

After the first three years of the program, 64 different Aboriginal and northern communities have received funding. This means the target, more than 50 communities by March 2011, has been exceeded. Of the 76 projects that received EANC funding between 2007–08 and 2009–10, one community received funding for six projects, one for three projects, eight communities received funding for two projects, one project funded involved four separate communities, and the remaining 50 communities all received funding for one project. Ontario and British Columbia have received almost 80 percent of EANC funding for more than 75 percent of the projects due to First Nation capacity and the number of communities that applied.

Of the 76 projects funded by the EANC in the first three years of the program, 44 (58 percent) of them were managed by Aboriginal and northern communities. This implies that the target of about 10 projects per year has been met or exceeded. The percentage of projects managed by Aboriginal and northern communities in the early years of the program increased significantly in 2009–10, from 48 percent in 2007–08 and 2008–09 to 73 percent in 2009–10.

Many key informants agreed that the EANC has improved Aboriginal and northern technical and management skills for clean energy. Almost all program recipients interviewed confirmed the key informants' statements indicating that their projects have resulted in the teaching and training of people within the community to develop the skills to operate and maintain clean energy technologies. This includes the ability to operate and maintain small hydro facilities and district heating systems, monitor and analyze demand-side energy management systems, and develop the skills for proper energy-efficient usage. The case studies indicated that the training came directly through the work done on the project, from third-party instruction, or from the attendance of clean energy conferences. Some key informants noted that the improvements in technical and management skills for clean energy are happening slowly and gradually, and several communities are not there yet.

Some of the other key informants dispute that the program is improving Aboriginal and northern technical and management skills for clean energy and that it is outside the capability of the government to achieve such a result. Therefore, while the EANC may contribute to the enhancement of technical and management skills, these respondents are sceptical that the program is driving this process. It is not possible to assess this rigorously in the absence of a systematic survey of Aboriginal community leadership.

Successfully implemented renewable energy and energy efficiency projects

After the first three years of the program, 55 renewable energy projects and eight energy efficiency projects were funded. Of these, six renewable energy and four energy efficiency projects had been commissioned as of March 31, 2010, and it is expected that another eight renewable energy and five energy efficiency projects will be commissioned by the end of the program on March 31, 2011. [Note 35]

Of the five renewable energy projects examined in the case studies, two have been successfully implemented and three have not begun construction. For the three renewable energy projects that have not begun construction, the EANC funding was used to support the environmental assessments and other early phases of the projects. Of those three projects, two still need to leverage a lot of funding for their projects to proceed to the construction phase.

Of the two energy efficiency case studies, those projects were successfully implemented and have been completed. The community energy plan case study project has already led to the installation of a 10-kilowatt (KW) solar photovoltaic (PV) system in the community.

Almost all key informants agreed that the program has successfully implemented renewable energy projects, and about half of them agreed that the program has successfully implemented energy efficiency projects. Only some key informants indicated that no renewable energy and/or energy efficient EANC-funded projects had been constructed in their region.

Intermediate outcomes

Many intermediate outcomes have been achieved.

Aboriginal and northern communities are equipped with knowledge and tools to increase energy efficiency of community infrastructure

The indicator for this intermediate outcome is the same as for the immediate outcomes. In order to better assess the achievement of this intermediate outcome, qualitative information was collected through case studies and key informant interviews.

Many key informants and all of the program recipients interviewed agreed that Aboriginal and northern communities have become equipped with the knowledge and tools to increase the energy efficiency of community infrastructure. Some key informants attributed the achievement of this outcome to community energy plans; however, a few of them indicated that the energy plans were funded under ANCAP and not the EANC. A few key informants said that this outcome had not yet been achieved.

Program recipients interviewed said that the program increased awareness of clean energy in the community and the surrounding communities, increased awareness nationally of the communities implementing the projects; and changed energy use behaviour of residents from increased awareness through the projects.

The community energy plan and the energy efficiency case studies provided the following direct evidence for this outcome:

• The community energy plan case study provided information to the community about improving the energy efficiency of community infrastructure and, in some cases, projects stemming from the community energy plan are close to being implemented.
• The community that installed the smart meters developed the skills to monitor their energy usage and lower it as necessary.
• The customers receiving the residual heat from the district heating system were trained on how to properly use and administer the residual heat.

Decrease in emissions of GHGs and CACs in northern and Aboriginal communities

The target estimated reductions of GHG emissions resulting from the EANC is 1.3 Mt over the life cycle. [Note 36] The estimated reductions are calculated from project proposals submitted by proponents. In the proposals, proponents are required to provide expected GHG emissions reductions from the projects and it is recommended that they submit a RETScreen analysis. Over the first three years of the program, projected GHG savings expected as per proponents are .655 Mt/year and 13.04 Mt over the life cycle of the projects funded by the EANC. Of the 63 renewable energy and energy efficiency projects, [Note 37] seven did not provide projected GHG savings estimates; therefore, should these seven have provided their expected GHG savings, then the overall expected estimated life cycle GHG savings might have been higher than the 13.04 Mt.

All proposals are subject to a technical review by a third party, who examines the information provided by the proponent by using technical standards and industry norms to see if there is a need to adjust energy assumption provided by the proponent. Those reviewed estimates are used to calculate the overall expected GHG emissions reductions from the program. The third party reviewed 45 of the 63 renewable energy and energy efficiency EANC-funded projects. Third party estimates GHG savings of .271 Mt/year and 5.41 Mt over the life cycle of the projects. An EANC representative noted that due to limited operation and maintenance funds, third-party reviews are not done on a project-by-project basis. Third-party estimations of reviewed projects are lower in average by more than 50 percent than proponent estimations and the median is 78. In 64.5 percent of the cases, the third-party projections are lower than the proponent projections, 13.3 percent equivalent and 22.2 percent higher.

Both the proponents' and the third parties' GHG reduction estimates over the 20 years of the projects are above the program's expected reduction of 1.3 MtGHG emissions.

The EANC only calculates GHG emissions reductions based on expected data, not monitored data, [Note 38] as few projects have been completed yet.

The expected GHG emissions reductions of the six renewable energy and four energy efficiency EANC projects that have been commissioned as of March 31, 2010, appear in Table 5.

Source: INAC (no date). Regulatory and ecoACTION Program Analysis - Report in Response to the Kyoto Protocol Implementation Act 2010.

There was no indicator and target for CACs. It should be mentioned, however, that Canada's record on CACs is much better than its record on GHGs. Nationally, emissions of CACs have declined over the 1990–2007 period. (EC, 2007)

Increased energy efficiency and use of renewable energy in northern and Aboriginal communities

In total, 33 of the 55 renewable energy projects that received EANC funding in the first three years of the program provided estimates of the amount of clean energy they will generate per year. The 33 projects are estimated to generate 1,012,289.8 megawatt hours (MWh)/year of clean energy. As this was the only quantitative measure available from the EANC's performance measurement system, the evaluation cannot validate this intermediate outcome.

More reliable energy infrastructure in northern and Aboriginal communities

The evaluation cannot appropriately assess this intermediate outcome since the program has not measured this; qualitative information is however used to provide findings. Almost all program recipients interviewed and many key informants agreed that the program had achieved this outcome or was leading towards achieving it. They indicated that the construction or planned construction of the renewable energy and energy efficiency structures would provide more reliable infrastructure for the community.

However, some key informants agreed that more reliable infrastructure in northern and Aboriginal communities was not achieved through the EANC. They said that the off-grid communities are still relying on diesel energy and that the funding available through the EANC is not enough to make a significant difference to overall fossil fuel use.

Energy cost savings in northern and Aboriginal communities

The projected energy cost savings were only provided for nine projects in 2009–10 in the EANC's performance measurement system. The total projected energy cost savings of the nine projects is $842,343/year. No projected energy cost savings data were provided for the first two years of the program, which included seven energy efficiency projects. All EANC supported energy efficiency projects are expected to result in energy cost savings.

Some of the case studies demonstrated energy cost savings from the implementation of the projects:

• The two energy efficiency case study projects led directly to reduced energy costs for Aboriginal and northern communities. The smart meter program resulted in cost savings for residents in the form of lower diesel consumption to produce energy, and the district heating system lowered heating costs by 10 percent for those participating in the system;
• While four of the renewable energy case study projects will not result in direct energy cost savings for the communities, the solar installation in the community will, by reducing certain buildings' reliance on power from the grid. The annual power savings for the community project were calculated at $10,652.69 per year by comparing the current power produced and the historical load based on the previous year's Hydro statements.
• Although the community energy plan case study project will not result in direct energy cost savings to the community, the projects stemming from the plan, such as the 10-KWPV solar installation, will result in energy cost savings.

Many key informants agreed that the EANC was either resulting in energy cost savings in northern and Aboriginal communities or leading toward this outcome. While a few key informants noted that energy cost savings in these communities was the main benefit of the implementation of the energy efficiency projects, others were not sure if the program was achieving this outcome.

Long-term outcomes

The EANC activities are leading to the achievement of most of the long term outcomes.

Reduced GHG emissions and CACs in Aboriginal and northern communities (1.3 mega tonnes of GHG reductions over the life cycle of projects supported by the initiative)

As indicated under the second intermediate outcome, over the first three years of the program, projected GHG savings over the life cycle of the projects funded by the EANC as estimated by the proponents is 13.04 Mt and as estimated by the third parties is 5.41 Mt. As it includes many studies, assessment and design projects, those projects do not contribute to reduction except when the next steps or phases are being implemented. However, according to the estimations, the expected GHG reductions are higher than the target of the program of 1.3 Mt.

Almost all key informants agreed that the EANC is and will result in reduced GHG and CACs emissions by selling energy directly into the power grid, therefore reducing the regions need to produce or import 'dirty' energy, and by reducing the off-grid communities' reliance on diesel fuel. However, a few key informants indicated that the EANC was not a large enough fund to reduce the communities' reliance on diesel and therefore the program was not having as big of an effect on GHG and CACs emissions as desired.

The program collected no CACs data.

Sustainable and healthy northern and Aboriginal communities

Many key informants and all of the program recipients interviewed agreed that the program is leading towards this outcome. They mentioned that this is the main goal of implementing these projects in Aboriginal and northern communities. The program recipients interviewed indicated that their projects will lead to this outcome through increased job creation in the community during the construction and operation of the clean energy technologies, inject millions of dollars into the communities through selling the renewable power to the grid, and, in the energy efficiency cases, put more money directly into residents' hands, and increased business development/expansion. They also indicated that the reduction of diesel energy use in the community from the EANC-funded projects would reduce pollution in the communities and decrease related health issues. At this point, there is no updated inventory of energy use, which could act as a national baseline data.

One of the case studies provided a direct example of how ecotourism, an economic spin-off from their renewable energy project is helping make their community more sustainable. Individuals from all over Canada and the world have come to see the project resulting in financial gains.

In addition, many of the communities are "off-grid" and rely on diesel generators for electricity generation. One of the main environmental benefits from renewable energy investments for these off-grid communities will be their decreased reliance on diesel. Another benefit will be reduced exposure to price fluctuations in oil and avoidance of the potential for sustained price increases in the future. The communities' reduced reliance on diesel will increase the financial viability of the communities and reduced emissions. However, some key informants noted, that the EANC does not offer sufficient financial support to remove these communities from reliance on diesel. The renewable energy and energy efficiency technologies installed under the program only supplement the energy provided by the diesel generators, and therefore, to the extent that they continue to need diesel-based electricity generation, these communities are still susceptible to oil price fluctuations and CACs emissions.

B1.8 To what extent were program funds leveraged to ensure effective implementation of projects?

The EANC provides early funding for projects. Almost all the key informants and the program recipients interviewed specifically indicated that the important aspect of the EANC funding was its timing more than the amount. The respondents said that the EANC often provides dollars for the early stages of projects and helps leverage dollars from other funders. They noted this was especially true for large renewable energy projects, where $250,000 is a minimal dollar amount compared to the overall costs of the projects.

In other words, from the quantitative evidence available, the EANC has had divided success in leveraging additional funds for projects. In the first three years of the program, the EANC provided just under $7.5 million in contributions to projects. The total estimated cost of all of these projects is more than $200 million. Therefore, more than 26 times the EANC's contribution has been put into these projects from other funders or from the communities putting in their own dollars.

The case studies provided evidence of both successfully and unsuccessfully leveraged funding for projects. Some case studies showed that the EANC resulted in leveraging additional funds for their projects. Many case studies only received funding from the EANC and therefore, did not need to leverage additional dollars for that aspect of their project; however, some of these projects are now trying to leverage dollars to begin construction, and some have been unsuccessful up to this point. In these cases, the EANC funding did not lead to the leveraging of funds.

B1.9 To what extent has the program contributed to constructive interaction and collaboration between the utilities and Aboriginal and northern communities?

The EANC is resulting in constructive interaction and collaboration between the utilities sector and Aboriginal and northern communities, according to almost all key informants and program recipients interviewed. The EANC is laying the foundation for future partnerships, which has emerged as one of the main benefits of this program.

According to the interviewees, the EANC provides communities with the capacity and ability to engage the utilities sector. For example, on-grid communities looking to sell renewable power to the grid need to develop an agreement to sell the power to the utilities companies. The respondents also indicated that the EANC has encouraged the utilities sector to engage the communities. The primary regional utilities sector is interacting more with the communities in remote areas in the interest of selling power to those diesel environments. However, few key informants identified P/T energy policies and programs as instrumental in creating this constructive interaction.

4.4 Unintended Outcomes

The unintended outcomes mentioned by key informants and program recipients interviewed were almost all positive. The most commonly mentioned unexpected outcome was the development of relationships and partnerships with the utilities sector and other important groups.

B1.10 What unintended outcomes (positive or negative) have emerged from the implementation of the program?

The most commonly mentioned was the development of relationships and partnerships. Specific comments made by key informants included:

• The development of relationships with neighbouring communities and municipalities through the sharing of information and benefits; and the building of relationships with P/T utility companies and governments.

Others mentioned by key informants and program recipients interviewed in order of frequency included:

• The development of communities into "real" players in the energy industry and the bringing of communities together to rally around a project. Also, additional projects stemmed from EANC-supported projects.

One unintended outcome mentioned by some of the INAC key informants was that the EANC has resulted in increased demand for provincial funding. However, communities are not applying for provincial funding since these provincial programs are expending all of their funds.

A couple of negative unintended outcomes mentioned by program recipients included:

• Difficulty of getting projects under way with the utilities sector and difficulty working with outside financial institutions.

5. Evaluation Findings - Performance (Lessons Learned and Best Practices)

Lessons learned from the evaluation mostly revolved around the importance of the regional staff and the importance of community energy plans to achieve the objectives of the EANC.

B1.11 Were there any best practices or lessons learned from the program? (Note: this also answers question B1.2)

The key informants and program recipients addressed lessons learned from the EANC. These, and the best practices mentioned by the respondents, in order of frequency, included:

• The regions that have staff to dedicate resources to the program appear to have had more success getting projects funded and initiated;
• Communities who developed a conservation plan before embarking on a large renewable energy project reduced energy use and increased efficiency at much less cost and tended to have an immediate direct impact on reducing GHGs and CACs;
• Having an ongoing working relationship with INAC eased the application process and helped identify possible solutions in the community to address energy issues in the region; and
• Working with and getting approval for the project from First Nation leadership is often a prolonged process, which affects the implementation timeline.

6. Evaluation Findings - Performance (Economy and Efficiency)

The economy aspect of the EANC was not addressed, as it required financial information for each output of projects and operating costs in order to estimate the economy of resources used. These data are not available. The program's efficiency and cost-effectiveness were difficult to determine. While comparisons can be made with similar programs, often the information from other programs is not available most particularly because the programs are inherently different from each other.

Although many key informants were unsure of whether the EANC was cost-effective at reducing GHGs and CACs, program recipients interviewed suggested that the program is cost-effective because it gets good return on its small investments in multi-million dollar projects.

B2.1 Is the program within its budget? If not, what accounts for the variances (positive and negative)?

The total actual program spending is within budget. Overall program spending per fiscal year has remained fairly close to expected expenditures. While the program spent less than expected in its first year when it was just starting, it spent slightly more than expected in the following two fiscal years. The grants and contributions of the program have followed a similar pattern to overall spending. The program has $3,782,471 to spend in 2010–11, which is approximately $30,000 more than expected at the outset of the program (see Table 1 and Table 2 in subsection 1.2.3).

B2.2 Does EANC provide a cost-effective method to reduce GHGs and CACs?

The evidence from this evaluation does not allow for conclusion as to whether this program represents a cost-effective method of reducing GHGs and CACs. In theory, comparisons of the cost-effectiveness of this program could be made with others; however, retrieving this information from these other programs is difficult and furthermore, doing these types of comparisons is risky because the programs are inherently different.

The program does not collect cost saving information from energy efficient project completion reports. The bottom-up models such as RETScreen offer the possibility of comparing the projected GHG reduction. Dividing this by the capital and operating costs of a technology offers a basis for cost-effectiveness analysis within the program. However, information on operating cost is not available. Table 6 shows projected GHG savings per dollar of EANC funding spent on projects. Since not all the projects provided GHG savings estimates, the calculations included dividing the total projected GHG savings by the total EANC funding amount for those projects only. Table 6 also shows that the projected GHG savings per dollar of EANC funding spent is decreasing year after year since 2007–08 [Note 39], which indicates that benefits are bigger the first years.

It is theoretically possible to compare the cost per GHG tonne reduced under the EANC with other federal approaches such as ecoENERGY for Equipment, ecoENERGY for Personal Vehicles, ecoENERGY for Fleets, and ecoENERGY for Transportation (all NRCan) when recent evaluations are released. [Note 40] However, currently comparisons cannot be made to other programs since the required information is not available.

In general, renewable energy alternatives such as geothermal, wind, and solar are evolving as is their cost-effectiveness. These renewable energy alternatives presently require that conventional energy generation be maintained as a supplemental resource, which decreases the cost‑effectiveness. However, as per case studies, another option was to increase the energy consumption awareness in northern and Aboriginal communities to decrease the amount of energy generated from harmful sources such as diesel plants. Many key informants were unsure of whether the EANC was cost-effective at reducing GHGs and CACs.

Many program recipients interviewed said the program is cost-effective because it provides funding for the early, risky phase of the project and that helps projects to both get off the ground as well as leverage other dollars (see B1.8). However, few projects were unable to leverage the total funds to move to the implementation phase.

7. Evaluation Findings - Performance (Alternatives)

In the absence of any specific alternative compatible with Aboriginal capacity, it is important to reiterate the significant concerns expressed by interviewees: close cooperation and joint funding of projects and initiatives in other federal departments and other orders of government and the decentralizing of the program to be delivered at the regional level.

B2.3: Are there alternative approaches that are just as or more efficient and/or effective at reducing GHGs and CACs in Aboriginal and northern communities as/than EANC?

In the absence of any specific alternative that is compatible with Aboriginal capacity, the main alternatives, noted from the literature, involve close cooperation and joint funding of projects and initiatives by other federal departments and other orders of government. For example, managing the demand for fossil fuels by increasing insulation may have a high pay-off in the North, but cooperation with Canadian Mortgage and Housing Corporation and P/T governments is necessary in such an approach. Case studies demonstrated funding involvement with other federal departments and other orders of governments.

The P/T governments are active in a wide range of "green" projects. As an example, initiatives by the Yukon government to extend the electrical grid and create small-scale hydro projects may be more cost-effective than the investment in renewable energy technologies such as solar and geothermal. However, such an analysis is beyond the scope of this evaluation, but would be a worthwhile study.

Some key informants suggested alternative approaches to deliver the program, including the more commonly mentioned approach of decentralizing the program and having it delivered at the regional level. Some key informants indicated that regional staff is more familiar with the communities and it would be more efficient for them to dispense the funding than having to send the proposal to headquarters for approval. In this kind of alternative headquarters would still be responsible for policy direction, but the administration of the program would be at the regional level.

8. Conclusions and Recommendations

8.1 Conclusions

The purpose of the evaluation was to provide evidence and conclusions regarding the relevance and performance of INAC's delivery and implementation of the EANC. The evaluation examined EANC activities undertaken between April 1, 2007, and March 31, 2010.

The evaluation supports the following conclusions regarding relevance and performance:

Relevance

The EANC is a relevant federal program under its main current goal of reduced GHG and CACs emissions. It supports both the federal government priorities and INAC's strategic objectives. INAC has an appropriate role in clean energy projects due to its responsibilities, and its experience with Aboriginal and northern communities. The provinces/territories have jurisdiction over energy generation. However, the EANC's funding for these projects is still relevant as there are few other funding sources available for these types of projects in Aboriginal and northern communities.

Performance

Effectiveness (i.e., Success)

While most of the planned EANC activities have been implemented to some extent, all of the performance objectives that could be assessed quantitatively were but many did not have targets. As a result, measuring the extent of their success proved difficult. Furthermore, data was not collected on the main outcomes of the CACs. There were too many indicators that were repetitive, particularly at the output level and did not provide the right level of information.

The quantitative and qualitative data available showed that the EANC has been delivering its outputs, which are leading towards the achievement of its expected outcomes. In the cases where targets were available, the EANC appeared to have achieved its outputs and outcomes, except for its expected support of community energy planned projects, where it fell short. Collaboration between the utilities sector and Aboriginal and northern communities, and the development of those partnerships, while not an expected outcome identified at the start of the EANC, became one of the main benefits of the program.

A key feature of the EANC is its provision of early funding to get projects started. This allowed for the leveraging of twenty-six dollars per each dollar invested by EANC, from other sources, which supported the successful completion of EANC funded projects. However, leveraging of dollars for the successful completion of EANC funded projects has been mixed, as not all the projects were able to progress to implementation due to lack of funding. Other projects were funded solely by INAC, therefore with no leveraging.

Lessons Learned and Best Practices

Key informants and program recipients interviewed provided few suggestions related to lessons learned and best practices. These suggestions revolved around the importance of dedicating regional staff to the program, having ongoing working relationships with INAC staff, and developing community energy plans before embarking on large renewable energy and energy efficiency projects.

Economy and Efficiency

The economy aspect of the EANC was not addressed, as it required financial information for each project's output and operating costs in order to estimate the economy of resources used. These data are not available, as the program does not collect this information.

The program's efficiency and cost-effectiveness were difficult to determine. While comparisons can be made with similar programs, information from other programs was not available. Further, the programs are inherently different from each other, making comparison less meaningful. Although many key informants were unsure of whether the EANC was cost-effective at reducing GHGs and CACs, program recipients interviewed suggested that the program is cost-effective because it gets good return on its small investments in multi-million dollar projects.

Alternatives

In the absence of any specific alternative that is compatible with aboriginal community capacity, the main alternatives, noted from the literature and identified in the case studies, involve close cooperation and joint funding of projects and initiatives in other federal departments and other orders of government. Some key informants noted that decentralizing the program delivery to the regional level could be beneficial for the program.

8.2 Recommendations

It is recommended that INAC:

1. Move towards better integration of renewable energy and energy efficiency considerations with other jurisdictions and with INAC's programs, such as the Community Economic Opportunities Program, Major Resource and Economic Development, Strategic Partnership Initiative within economic development, with community infrastructure, and with other related community development initiatives.
2. Clarify the overall direction of the program, determining whether it is intended to operate as a program to provide incentives for the launch of projects in aboriginal and northern communities (i.e. setting the groundwork) or to contribute to the achievement of eco‑energy objectives.
   1. Refocus the performance measurement strategy to strengthen the capacity to collect the appropriate data on the results of both the projects funded and the program.
   2. Align administration and reporting requirements to the amount invested and the level of risk (program and recipient).

Appendix A - EANC Logic Model

The logic model outlines how the activities align with the expected outputs and outcomes of the EANC. The following figure shows a logic model for the ecoEnergy for Aboriginal and Northern Communities program. There are four activity areas, as follows:

1. Communications, networking, and outreach activities—targeted to Departmental Capital Program Staff in the regions and external stakeholders;
2. Procedural support for the acceptance, review, and approval of projects;
   
   
3. Technological and financial advice on the application of renewable energy and energy efficiency technology in northern and Aboriginal communities; and
4. Technical review, assessment, and approval of projects.

These four activity areas are linked via arrows to the six outputs associated with these activities, which are as follows:

1. Communication products and promotional materials
2. Established 'clean energy network'
3. Advice on technical feasibility and project management skills to proponents
4. Renewable energy projects are funded in accordance with program objectives
5. Energy efficiency projects are funded in accordance with program objectives
6. Community energy plans are funded in accordance with program objectives

The outputs are linked via arrows to three immediate outcomes (3 years), which are as follows:

1. Northern and Aboriginal communities are engaged and participating in the ecoENERGY program
2. Improved northern and Aboriginal technical and management skills for clean energy
3. Successfully implemented renewable energy and energy efficiency projects

These immediate outcomes for outputs are, in turn, linked via arrows to five intermediate outcomes (3-5 years), which are as follows:

1. Aboriginal and northern communities are equipped with knowledge and tools to increase energy efficiency of community infrastructure
2. Decrease in emissions of GHGs and CACss in northern and Aboriginal communities
3. Increased energy efficiency and use of renewable energy in northern and Aboriginal communities
4. More reliable energy infrastructure in northern and Aboriginal communities
5. Energy cost savings in northern and Aboriginal communities

These intermediate outcomes are then linked via arrows to two long-term outcomes (5+ years), which are as follows:

1. Reduced GHG emissions and CACss in Aboriginal and northern communities (1.3 mega tonnes of GHG reductions over the life cycle of projects supported by the initiative)
2. Sustainable and healthy northern and Aboriginal communities

Appendix B - Evaluation Matrix

Appendix C - EANC Performance Measurement Strategy

Appendix D - Terms of Reference - March 2010

1.0 Overview

Indian and Northern Affairs Canada (INAC) will carry out an impact evaluation of the ecoENERGY for Aboriginal and Northern Communities (EANC) Initiative.  This evaluation will provide evidence-based conclusions regarding relevance, design and delivery, results/success, and cost-effectiveness/alternatives of INAC's approach towards providing assistance to Aboriginal and Northerners (Aboriginal) as it pertains to the EANC. It will support renewal of contribution authorities in 2010-2011.

1.1 Evaluation Requirement

The Clean Air Agenda (CAA) Horizontal Management, Accountability and Reporting Framework (HMARF) architecture is a horizontal Program led by Environment Canada (EC), involves eight (8) federal departments and agencies and consists of forty-three (43) programs organized into eight (8) themes based on common program-level objectives:

• Clean air regulations (9 programs);
• Clean energy (7 programs);
• Clean transportation (12 programs);
• Indoor air quality (2 programs);
• International actions (4 programs);
• Adaptation (7 programs);
• Partnerships (1 program); and,
• Management and accountability (1 program).

Partner departments and agencies are responsible for evaluating their respective programs from which the evaluation results will be rolled up to the Clean Energy Theme (CET) level. Natural Resources Canada (NRCan) is the CET lead and as such is responsible for the completion of the thematic evaluation in fiscal year 2010-2011.

In effect, the INAC evaluation results will be rolled up at two levels for the CET: from the initiative-level evaluation to thematic evaluation reports and from the thematic evaluation reports to the overall CAA Horizontal evaluation.

2.0 Program Description

2.1 Background

The April 2006 Speech from the Throne committed the government of Canada to "take measures to achieve tangible improvements in our environment, including reductions in pollution and greenhouse gas (GHG) emissions."  That commitment led to the CAA which resulted, in November 2006, in an approved investment of $2,061,000,000 from 2007-08 to 2020-21 to increase renewable energy and energy efficiency. The CAA is a set of environmental initiatives and programs, coordinated and delivered by the 8 departments and agencies with EC as the lead. The nine departments and agencies (EC included) participating in the CAA are:

• Environment Canada
• Natural Resources Canada
• National Research Council
• Transport Canada
• Industry Canada
• Health Canada
• Foreign Affairs and International Trade
• Public Health Agency of Canada
• Indian and Northern Affairs Canada

Clean Energy is one of the eight themes of the CAA and encompasses a comprehensive strategy to reduce emissions from both production and use of energy and, is designed to complement the regulatory aspects of the CAA. In addressing clean air and climate change for the four-year period, the CAA involves the following two Initiatives:

1. Clean Air Regulatory Agenda ($0.35 billion), and
2. Clean Air Program Measures (CAPM) in support of the CAA ($1.35 billion).

Of the second Initiative (i.e.CAPM), INAC's role is to support the government's commitment to assist Aboriginal and Northern communities in fulfilling their aspirations for greater self-reliance. Hence the EANC serves to support First Nations and Northern communities wishing to engage in Renewable Energy and Energy Efficiency projects by providing supplemental financial resources needed to assess, develop and implement more energy efficient and renewable energy technologies and projects.

2.2 Program Objectives and Expected Outcomes

The EANC contributes to INAC's Strategic Economy Outcome. It's objectives (see Logic Model, Annex A) are to facilitate renewable energy and energy efficiency projects that will:

• Reduce GHG emissions by a projected 1.3 mega tonnes over the life cycle of projects (defined as 20 years) supported by the EANC; and
• Decrease Criteria Air Contaminant (CACs) emissions resulting in improved air quality.

These objectives are expected to be achieved through funding of projects that reduce or displace natural gas, coal and diesel generation of electricity through more energy efficient technologies and increased use of renewable energy. Expectations are that successful projects will produce other social, environmental, and economic development benefits for communities participating in the EANC.

In addition, the EANC is expected to equip Aboriginal and Northern communities with appropriate knowledge and tools to increase the energy efficiency of community infrastructure, to access renewable energy opportunities and to implement cost effective renewable energy projects. The EANC provides a legitimate and necessary role for INAC (and the federal government) to assist Aboriginal and Northern communities with energy project needs, where capacity for these types of projects is limited.

The EANC objectives are consistent with, and are expected to also contribute to a number of departmental objectives, such as:

• Providing assistance to Aboriginal and Northern communities to invest in physical assets that mitigate health and safety risks;
• Ensuring that activities are undertaken in an environmentally sound and sustainable manner; and
• Equipping Northerners, their governments, and a number of organizations and stakeholders, as well as FN and Inuit communities and organizations, with the ability to carry out activities which complement departmental efforts related to safe use, development, conservation, and preservation of the North's natural resources and the implementation of northern contaminants, climate change and sustainable development strategy initiatives.

The EANC provides funding support directly to Aboriginal and Northern communities to assist with eligible clean energy projects where they are technologically and financially feasible. The focus is on obtaining concrete results for the strategic federal investment in clean energy projects. These energy projects fall into three categories that are designed to support proven energy technologies (excluding demonstrative and pilot technologies or projects):

• Renewable energy: small or micro hydro facilities, wind turbines, solar electric, and biomass, etc.
• Energy efficiency: improved diesel generation efficiency (in off-grid communities), district heating applications, solar thermal, improved energy/heating systems in community buildings, etc.
• Community Energy Plans: the program will provide technical and financial support to eligible organizations to develop action plans that address clean energy in their community.

It is anticipated that the program will support approximately 32 to 55 projects per year, broken down as follows:

• Renewable Energy and Energy Efficiency – 6 to 15 projects directly funded per year,
• Energy Efficiency - 10 to 20 infrastructure projects that are in INAC's Long Term Capital Plan, and,
• Community Energy Plans – 16 to 20 per year.

2.3 Program Management, Key Stakeholders and Beneficiaries

The design and approach to delivery of the EANC builds on the success that INAC has had with other programs of a similar scale and complexity, such as the 2003-2007 Aboriginal and Northern Community Action Plan (ANCAP), the first Federal program fully dedicated to engaging and involving Aboriginal communities in climate change action. ANCAP aimed to assist Aboriginal and Northern communities to develop initiatives in community energy planning, renewable energy (RE), alternate diesel and technologies, etc. The following roles and responsibilities of the EANC were developed based upon previous experience and lessons learned from the delivery of ANCAP.

Central (HQ) delivery: The EANC is centrally delivered by staff in the Environment and Renewable Resources Directorate (ERR) in Northern Affairs who review applications, ensure the eligibility of the proponent, and approve eligible projects.

Together with the Director, ERR, the EANC is administered by HQ staff, including provision of technical advice to applicants, supporting the Project Review Committee (which, among other things, is responsible for reviewing all proposed projects, ensuring that recommended projects are feasible and meet eligibility criteria, and providing a recommendation to the Director on whether or not to approve the project), monitoring project implementation, providing support to recipients and also working indirectly with external stakeholders (e.g. provincial and territorial government energy departments) to ensure effective project implementation.

Project Review Committees: comprises of representatives from key INAC sectors (Economic Development, Capital Programs and Northern Affairs) and a Technical contractor (third party) who consider all eligible projects and recommend, for the approval of the Director, those projects which it considers most appropriate for funding. Where recipients of approved projects are at the time receiving funds through Contribution Agreements (CAs), those agreements are to be amended to include the EANC funding. Where recipients are not, at the time receiving funds through a CA, then a new agreement is to be negotiated for the purposes of providing the EANC funding. Project officers located at Headquarters (HQ) are to monitor the implementation of agreements; however the agreements are to be managed in the specific INAC region in which they are issued.

External Stakeholders: Program staff (HQ) are also expected to work indirectly with external stakeholders (e.g.the utilities sector) to ensure effective project implementation.  External stakeholders are essential to the success of the EANC for a number of reasons, including:

• Since the ecoENERGY Initiative only partially funds a project, other funding partners are required to successfully implement projects, and
• The success of many projects will depend on selling renewable energy to the provincial power utilities through integrated grid ties, providing for important project revenues that make many of these projects financially feasible.

Recipients: responsible for project implementation, in accordance with approved project plans submitted (with applications), and with the terms and conditions provided in the CA.

2.4 Program Resources

Treasury Board approved $15,000,000 in total funding over the four-year period from 2007-2008 to 2010-2011. Starting in 2007-08, Table 1 below provides a breakdown of annual resource allocations and expenditures for the EANC (in millions $):

Costs associated with monitoring and oversight activities are included in the operating vote (Salary and O&M), and are as follows:

Resources were to be allocated to approved projects through CAs at the community level. Any EANC funded projects will be included in existing agreements that INAC has in place with Aboriginal communities. Where a recipient does not have a current CA, then such an agreement will be prepared for program funding.

The EANC is to fund projects to the following levels for the three major types of clean energy projects [Note 41]:

• Renewable energy projects – up to $250K per project;
• Energy efficiency projects – up to $250K per project; and,
• Community energy plans – $5 to $15K, depending on the size of community.

3.0 Evaluation Methodology

3.1 Evaluation Objectives and Scope

The evaluation objectives are to examine the following:

• Planning, design, management, and delivery of the EANC
• Performance measurement processes and reporting
• Effectiveness of the EANC in funding relevant projects
• Extent to which the EANC is on track to meeting planned outcomes
• Program contributions to Clean Energy Theme and CAA outcomes

A number of evaluation questions are predetermined by the CET Evaluation Plan (see Annex B) to facilitate the roll up of CET evaluations which will focus exclusively on evidence that supports the CAA horizontal evaluation questions. Evaluation questions specific to the EANC will be included.

The evaluation will examine all EANC activities since April 1, 2007/08 to March 31, 2010. This will include the management review undertaken in 2009 which focused on the implementation of the EANC from fiscal year 2007-08 (when the program was initiated) through to the end of fiscal year 2008-09, as well as any audits conducted or under way.

3.2 Evaluation Issues

Relevance

This evaluation will assess the continued relevance of the EANC and whether or not it is consistent with departmental and government-wide priorities. The study will explore if the EANC realistically addresses an actual need as well as the appropriateness of the federal government role.

Specifically, the evaluation will consider whether or not the EANC has been appropriately aligned to facilitate renewable energy and energy efficiency projects intended to help Aboriginal and Northern communities develop the capacity needed to reduce GHGs over the life cycle of projects supported by the EANC and decrease the emissions of CACs resulting in improved air quality.

Design and Delivery

The extent to which each of the EANC activities has been implemented, or is on track to being implemented, as planned, will be assessed. Specifically, outputs and the degree to which the target beneficiaries are being reached will be assessed.

The evaluation will also determine the extent to which the design and delivery of the program has met its objectives (i.e., adequacy of program design and delivery) and if it has had the ability to address strategic outcomes whilst enabling effective delivery. Lessons learned and or best practices and results from previous evaluations will be considered to assess whether or not they have been incorporated in the EANC's design and delivery.

Success/Results

The evaluation will assess the effectiveness of EANC activities and the utility of resources to date in achieving outputs and outcomes. Whether the EANC is within budget, or not, will be assessed also. Unintended outcomes (positive or negative) that have emerged to date will be explored.

While assessing if the program is making progress toward the achievement of its intended outcomes, the evaluation will address such fundamental areas as data collection, appropriateness of performance information collected and reporting and accountability, as these are key to program success.

Cost-effectiveness/Alternatives

This component of the evaluation will include identifying program costs and exploring whether there are alternative means to producing comparable program benefits. The evaluation will assess if the most appropriate and efficient means are being used to achieve outcomes relative to alternative design and delivery approaches.

Considerations, Strengths and Limitations

The CET Evaluation Plan developed by NRCan in consultation with INAC's Evaluation, Performance Measurement and Review Branch and Program managers will be used as a guide throughout the evaluation.

This is an impact evaluation. The EANC has been in existence since 2007 and is the last of the CET programs to be evaluated. As such, preliminary findings will be required for the theme level evaluation in late 2010.

As well, a number of target communities and beneficiaries of the EANC are common to those of NRCan's Clean Energy Program. As such, efforts will be made to streamline evaluation methodologies wherever possible to reduce reporting burden and increase efficiency and effectiveness.

A notable limitation is the newness of the EANC. Though it may be built along a similar predecessor program like the ANCAP, its newness as a program could be a limitation.

Evaluation Methods

Data Sources

Subject to further development in the detailed methodology and work plan, the evaluation findings and conclusions will be based on the analysis and triangulation of the following lines of evidence:

• Review of files and documentation at HQ and in the regions, including audits, evaluations, review reports, and surveys, etc.
• Review of published documentation/literature, including international work, studies, and research.
• Review of INAC performance and financial data: management/financial systems, databases, the implementation of Performance Measurement Frameworks (PMF), and any other relevant reports will provide information to assess the impact of the EANC.
• Key informant interviews and/or and focus groups will target key players such as INAC managers, other federal departments (e.g., NRCan,), territorial governments, national Aboriginal organizations, Northern organizations, Aboriginal and Northern Communities and where applicable, professional organizations, and universities.
  
  The key informants will be asked to describe their management practices, including planning process, implementation, and monitoring to identify possible improvements, successes and constraints. They will provide information on impacts of EANC components.
• Case studies will provide a more in-depth study of best practices/lessons learned to date.
• Peer review of draft documents: key reports produced during the course of the evaluation, including questionnaire drafts, report drafts, will be reviewed by professionals.

4.0 Project Management and Quality Control

The Evaluation, Performance Measurement and Review Branch (EPMRB) will direct and manage the evaluation and coordinate the theme-level evaluation work with NRCan. The evaluation will be contracted out and EPMRB will provide regular progress updates to the Environment and Renewable Resources Directorate of the Northern Affairs Sector.

Quality control measures will allow for the validation of evaluation products (e.g., technical reports and draft reports) throughout the evaluation. A Working Group may be set up consisting of EPMRB and EANC managers.

5.0 Evaluation Resources and Estimated Timeline

Estimated Budget and timelines
The estimated budget to conduct this evaluation is $100,000 and will be cost-recovered from the Program when the work begins. This budget could change (e.g., the estimated costs could be reallocated between expenditure types), as the detailed methodology and work plan are developed. The breakdown of the estimated budget is as follows:

Timelines:

Subject to verification in the detailed methodology report and work plan, the evaluation is expected to be completed by October 2010.

The Terms of Reference for the Evaluation of the ecoENERGY for Aboriginal and Northern Communities were approved by the Evaluation, Performance Measurement and Review Committee on May 14, 2010.

Annex A - Logic Model

The logic model outlines how the activities align with the expected outputs and outcomes of the EANC. The following figure shows a logic model for the ecoEnergy for Aboriginal and Northern Communities program. There are four activity areas, as follows:

1. Communications, networking, and outreach activities—targeted to Departmental Capital Program Staff in the regions and external stakeholders;
2. Procedural support for the acceptance, review, and approval of projects;
   
   
3. Technological and financial advice on the application of renewable energy and energy efficiency technology in northern and Aboriginal communities; and
4. Technical review, assessment, and approval of projects.

These four activity areas are linked via arrows to the six outputs associated with these activities, which are as follows:

1. Communication products and promotional materials
2. Established 'clean energy network'
3. Advice on technical feasibility and project management skills to proponents
4. Renewable energy projects are funded in accordance with program objectives
5. Energy efficiency projects are funded in accordance with program objectives
6. Community energy plans are funded in accordance with program objectives

The outputs are linked via arrows to three immediate outcomes (3 years), which are as follows:

1. Northern and Aboriginal communities are engaged and participating in the ecoENERGY program
2. Improved northern and Aboriginal technical and management skills for clean energy
3. Successfully implemented renewable energy and energy efficiency projects

These immediate outcomes for outputs are, in turn, linked via arrows to five intermediate outcomes (3-5 years), which are as follows:

1. Aboriginal and northern communities are equipped with knowledge and tools to increase energy efficiency of community infrastructure
2. Decrease in emissions of GHGs and CACss in northern and Aboriginal communities
3. Increased energy efficiency and use of renewable energy in northern and Aboriginal communities
4. More reliable energy infrastructure in northern and Aboriginal communities
5. Energy cost savings in northern and Aboriginal communities

These intermediate outcomes are then linked via arrows to two long-term outcomes (5+ years), which are as follows:

1. Reduced GHG emissions and CACss in Aboriginal and northern communities (1.3 mega tonnes of GHG reductions over the life cycle of projects supported by the initiative)
2. Sustainable and healthy northern and Aboriginal communities

INAC's mandate is to support Aboriginal people and Northerners in their efforts to:

• improve social well-being and economic prosperity;
• develop healthier, more sustainable communities; and
• participate more fully in Canada's political, social and economic development - to the benefit of all Canadians.

To this end, INAC's programs are currently delivered through six key Strategic Outcomes: The Government, The People, The Land, The Economy, The North and The Office of the Federal Interlocutor.

The EANC Initiative, when conceived, was designed to contribute to the departmental Strategic Outcome "The Economy" (Community Infrastructure).

At present, the EANC's planned outcomes contribute to the "The Economy" Strategic Outcome of the Program Activity Architecture (PAA) by directly supporting the PAA Sub-activity "Infrastructure Facilities" under "Community Infrastructure".

Annex B - CET Evaluation Plan

CET EVALUATION PLAN
(Developed by NRCan in consultation with INAC)

Relevance

• Are CLEAN ENERGY activities connected with key environmental needs?
• Are CLEAN ENERGY activities aligned with federal government priorities?

Design and Delivery

• Is the CLEAN ENERGY theme designed and delivered in a way that can best achieve program objectives?

Success/Results

• To what extent has the CLEAN ENERGY achieved its intended outcomes?
• Have there been unintended (positive or negative) outcomes? Were actions taken as a result of these?

Cost-effectiveness

• Are there more cost-effective and efficient means of achieving CLEAN ENERGY objectives?
• How could the efficiency of CLEAN ENERGY activities be improved?

Footnotes:

1. INAC. (2009). Operational Management Guide - ecoENERGY for Aboriginal and Northern Communities Program - 2007-2008 to 2010-2011. (return to source paragraph)
2. Treasury Board Canada Secretariat. (2008). The Clean Air Agenda. Retrieved August 25, 2009. (return to source paragraph)
3. INAC. (2009). Operational Management Guide - ecoENERGY for Aboriginal and Northern Communities Program - 2007-2008 to 2010-2011. (return to source paragraph)
4. Government of Canada. (2009). Theme: Clean Energy. Provided by program May 12, 2010. (return to source paragraph)
5. Government of Canada. (2009). Theme: Clean Energy. Provided by program May 12, 2010. (return to source paragraph)
6. Government of Canada. (2010). ecoENERGY - ecoENERGY for Aboriginal and Northern Communities. Retrieved May 4, 2010. (return to source paragraph)
7. An off-grid community is a community that is not connected to a power grid and therefore has to generate its own power. Often, the power is generated by diesel generators in these communities. (return to source paragraph)
8. INAC. (2009). Operational Management Guide - ecoENERGY for Aboriginal and Northern Communities Program - 2007-2008 to 2010-2011. (return to source paragraph)
9. INAC. (2009). Operational Management Guide - ecoENERGY for Aboriginal and Northern Communities Program - 2007-2008 to 2010-2011. (return to source paragraph)
10. INAC. (n.d.). Eligible Projects & Funding. [PowerPoint Presentation]; INAC. (2007). EcoEnergy for Aboriginal and Northern Communities Initiative Results-based Management and Accountability Framework and Risk Based Audit Framework. (return to source paragraph)
11. The funding amounts under the three categories are under the limits allowed under the two funding authorities: 334 (The People - Healthy Northern Communities) and 377 (The Economy - Community Infrastructure). (return to source paragraph)
12. INAC. (2007). EcoEnergy for Aboriginal and Northern Communities Initiative Results-based Management and Accountability Framework and Risk Based Audit Framework. (return to source paragraph)
13. INAC. (2009). Operational Management Guide - ecoENERGY for Aboriginal and Northern Communities Program - 2007-2008 to 2010-2011. (return to source paragraph)
14. INAC. (2009). Operational Management Guide - ecoENERGY for Aboriginal and Northern Communities Program - 2007-2008 to 2010-2011. (return to source paragraph)
15. INAC. (2007). EcoEnergy for Aboriginal and Northern Communities Initiative Results-based Management and Accountability Framework and Risk Based Audit Framework. (return to source paragraph)
16. INAC. (n.d.). Regulatory and ecoACTION Program Analysis - Report in Response to the Kyoto Protocol Implementation Act 2010. (return to source paragraph)
17. INAC. (n.d.). Regulatory and ecoACTION Program Analysis - Report in Response to the Kyoto Protocol Implementation Act 2010. (return to source paragraph)
18. The expected life cycle of a project is 20 years. (return to source paragraph)
19. INAC. (2007). EcoEnergy for Aboriginal and Northern Communities Initiative Results-based Management and Accountability Framework and Risk Based Audit Framework. (return to source paragraph)
20. INAC. (2009). Operational Management Guide - ecoENERGY for Aboriginal and Northern Communities Program - 2007-2008 to 2010-2011. (return to source paragraph)
21. INAC. (2007). EcoEnergy for Aboriginal and Northern Communities Initiative Results-based Management and Accountability Framework and Risk Based Audit Framework. (return to source paragraph)
22. INAC. (2007). EcoEnergy for Aboriginal and Northern Communities Initiative Results-based Management and Accountability Framework and Risk Based Audit Framework. (return to source paragraph)
23. INAC. (2007). EcoEnergy for Aboriginal and Northern Communities Initiative Results-based Management and Accountability Framework and Risk Based Audit Framework. (return to source paragraph)
24. INAC. (2009). Operational Management Guide - ecoENERGY for Aboriginal and Northern Communities Program - 2007-2008 to 2010-2011. (return to source paragraph)
25. Google Scholar is a specialized search engine that supports keyword searches of academic and technical literature. (return to source paragraph)
26. Seven of the eight case study proponents reviewed their case study write-ups and provided feedback to PRA Inc.(return to source paragraph)
27. Government of Canada. (2008). Speech from the Throne. Retrieved November 10, 2009; Government of Canada Privy Council Office.(2007). Speech from the Throne. Retrieved November 10, 2009. (return to source paragraph)
28. Government of Canada. (2009). Speech from the Throne. Retrieved July 29, 2010. (return to source paragraph)
29. Department of Finance Canada. (2007). Budget 2007. Retrieved November 10, 2009; Department of Finance Canada. (2008). Budget 2008. Retrieved November 10, 2009; Department of Finance Canada. (2009). Budget 2009. Retrieved November 10, 2009; Department of Finance Canada. (2010). Budget 2010. Retrieved July 29, 2010. (return to source paragraph)
30. INAC. (2007). EcoEnergy for Aboriginal and Northern Communities Initiative Results-based Management and Accountability Framework and Risk Based Audit Framework. (return to source paragraph)
31. INAC. (2009). Operational Management Guide - ecoENERGY for Aboriginal and Northern Communities Program - 2007-2008 to 2010-2011. (return to source paragraph)
32. INAC. (2010). Performance Measurement Summary - Excel Spreadsheet. Provided by program August 13, 2010. (return to source paragraph)
33. INAC. (2010). Performance Measurement Summary - Excel Spreadsheet. Provided by program August 13, 2010. (return to source paragraph)
34. Treasury Board of Canada Secretariat. (2009). Horizontal Initiatives. Retrieved July 30, 2010. (return to source paragraph)
35. INAC. (n.d.). Regulatory and ecoACTION Program Analysis - Report in Response to the Kyoto Protocol Implementation Act 2010. (return to source paragraph)
36. Expected life cycle of projects is usually estimated at 20 years.(return to source paragraph)
37. Community energy plans do not provide expected GHG savings. (return to source paragraph)
38. INAC. (n.d.). Regulatory and ecoACTION Program Analysis - Report in Response to the Kyoto Protocol Implementation Act 2010. Provided by program May 12, 2010.(return to source paragraph)
39. When other ecoENERGY evaluations are published, it should be possible to compare the cost per tonne of GHG reductions across a range of interventions. (return to source paragraph)
40. NRCan has the lead with respect to personal vehicles, fleets (buses and trucking), and transportation (federal vehicles). Transport Canada focuses on advancing technology to reduce fuel consumption. (return to source paragraph)
41. Where supplemental funding is being provided for energy related projects in the department's five year capital plan, then funding will be up to $100K (instead of $250K). These would be projects that will benefit from inclusion of supplemental funding to incorporate energy efficiency or renewable energy technologies. (return to source paragraph)