Evaluation of the Contaminated Sites On-Reserve (South of the 60th Parallel) Program

Relevance

The evidence demonstrates a continued need for the CSOR, given the number of contaminated sites and significant environmental site assessment and remediation work yet to be addressed (e.g., assessing and classifying suspected on-reserve contaminated sites). Such work will contribute to a decrease in priority sites and a reduction in risk and liability.^{Footnote 1} The need for the program will continue as new sites emerge. While the Federal Contaminated Sites Program (FCSAP), as a key source of funding supports the CSOR, the CSOR is well aligned with federal priorities (e.g., the promotion of a cleaner environment, and economic development). Alignment is also demonstrated by the additional Canada Economic Action Plan stimulus funding given to FCSAP, which was to enable expedited work on both the assessment and the remediation of federal contaminated sites. The expected result is to improve the environment while encouraging economic growth. All of these complement other INAC activities, such as community rejuvenation and the broader environmental and economic strategies of the federal government. Finally, the CSOR is consistent with federal roles and responsibilities, which includes legislative obligations related to environmental stewardship. It is also consistent with INAC responsibilities and community priorities such as reducing or eliminating contaminated sites and environmental hazards, as well as contributing to the safety of on-reserve communities.

Design and Delivery

Performance – Effectiveness

The CSOR is achieving its expected result that priority sites are remediated and risk managed and/or monitored. However, questions remain as to whether the program will be able to achieve full remediation, including closure of all contaminated sites. While the CSOR is enabling the reduction of liability, the total remediation liability for the CSOR is still prevalent due to the discovery and addition of new contaminated sites, which impact program-level progress on total liability reduction. There is also evidence to indicate that the secondary impacts of the program in the areas of employment and training are being achieved.

On the other hand, there is some evidence to indicate that secondary impacts of the program in the areas of employment and training are being achieved.

Performance – Efficiency and Economy

Evaluation evidence indicates that not all risks and financial liabilities associated with contaminated sites will be addressed given challenges, such as budgetary restraints. The constant discovery of new sites suggests a continuous demand for funding. Aside from FCSAP funds, the CSOR faces challenges with respect to leveraging internal INAC funds to address contaminated sites. This is compounded by the uncertainty surrounding whether the CSOR will obtain FCSAP funding beyond its expected end in 2020.

Despite these difficulties, as well as the requirement to demonstrate that the CSOR is spending its (i.e. public funds) in an efficient and economical way, and that it is achieving a positive impact relative to its capacity, the evaluation found that the CSOR has performed well in terms of resource utilization in relation to the production of outputs and progress towards its expected outcomes.

Some Key Challenges to Performance

Impact, both intended and unintended

There is general agreement among interviewees that the CSOR has had a significant positive impact in several areas, including a beneficial impact on First Nations overall. High priority (Class 1) sites are being remediated, pollution sources are being documented, and to the extent possible, awareness of the need for appropriate fuel handling and tank management procedures is being raised within on-reserve First Nation communities. It is expected that an increase in the community’s awareness will contribute to the program’s prevention efforts.

1.2.1 Background and Description

The CSOR is an environmental management program delivered by INAC and that funds First Nations to undertake environmental site assessment and remediation or to risk manage contaminated sites on-reserve and other lands under INAC (Lands and Economic Development) jurisdiction. These activities are supported to reduce or eliminate risks to human health and the environment, and to reduce legal and financial liabilities associated with contaminated sites. The CSOR is administered in accordance with INAC's Contaminated Sites Management Policy (2002) and the Federal Approach to Contaminated Sites developed by the interdepartmental Contaminated Sites Management Working Group^{Footnote 4}.

Implementation of the CSOR Program by INAC is supported by Environment Canada's Federal Contaminated Sites Action Plan Program (FCSAP) which provides funding for the CSOR on a cost-share basis. The FCSAP provides the majority of funding while INAC's Regional Operations'^{Footnote 5} Capital Facilities and Maintenance Program provides the balance. The Department's Lands and Economic Development Sector coordinates and flows funding to First Nations, liaises with the FCSAP Secretariat and Treasury Board Secretariat, and works with First Nations to develop and secure contracts for work to be undertaken on-reserve.

1.2.2 Overview of Contaminated Sites and Site Remediation Policy

1.2.3 Objectives and Expected Outcomes

The CSOR is expected to contribute to the expected result of community development by supporting the assessment and remediation of contaminated sites on-reserve lands and on any other lands under INAC's custodial responsibility. It supports the assessment and remediation of known contamination from the National Classification System for Contaminated Sites Class 1 and Class 2, for which a liability to the Crown has been established and documented. The expected outcomes for the Contaminated Sites On-Reserve are:

1.2.4 What classifies as an on-reserve contaminated site

Contaminated sites on-reserve land south of 60º are mostly related to abandoned dump sites and fuel spills^{Footnote 15}. These are sites that pose, or are likely to pose, an immediate or long-term hazard to human health and/or the environment and are often a legacy of past practices. If not managed properly, these sites, particularly those identified as having a higher risk to human health and safety, may have a significant impact on the ability for First Nations communities to capitalize on land-based economic development opportunities.

1.2.5 Program Management, Key Stakeholders and Beneficiaries

INAC is the federal department responsible for the delivery of Contaminated Sites On-Reserve.

1.2.6 Program Resources

Table 1 shows the operating expenditures and contributions of the CSOR from 2009-10 to 2013-14.

Note: While operating expenditures have been generally consistent between 2009–10 and 2013–14, contribution levels have varied year to year, with a large decrease in 2011–12.

Levels vary from year to year since they depend upon the program's ability to source funds within INAC and other departments, mainly FCSAP, Capital Facilities and Maintenance, Northern Contaminated Sites, and First Nations Land Management.

A contaminated site project is eligible for $10 million in funding annually. Where a project goes beyond this amount, approval (as in the case of a Capital project) is required from INAC's national headquarters. Projects are funded through contribution agreements between individual First Nations and INAC. The allocation and amount of funding are based upon an environmental assessment of the work required and depend upon various factors, including site classification score, human and/or ecological health receptors, type of land use (e.g., residential), the amount of available funds, legal obligations, economic development opportunities and cost-sharing opportunities. Depending on the region, payments are made monthly based on the recipient's cash flow forecast^{Footnote 24}. For example, in Manitoba, payments are made on the progress of the particular project and also on actual work done.

Table 2 shows the 2013–14 planned and actual program budget.

Note: The table is consistent with Table 1 and shows that the program spent over $15 million more than expected in 2013–14 due to the sub-program's ability to access additional resources from FCSAP and other sources^{Footnote 25}. Table 2 also shows that the program was able to operate in 2013–14 with about 4.6 less full-time equivalents than originally expected.

Funding for CSOR on-reserve comes from various sources: externally, from FCSAP and other federal departments (i.e. other FCSAP custodians); internally (i.e. INAC), from Lands and Economic Development, Regional Operations, the Northern Affairs Organization, and INAC Financial Management Committee requests.

Literature Review

The purpose of the literature review was to acquire a greater understanding of contaminated sites and approaches to determining, classifying and dealing with contaminated sites on-reserve land across various Canadian jurisdictions; and, examine the impact of contaminated sites on-reserve south of the 60th parallel. The literature was selected and reviewed on the basis of its relevance to the evaluation and it included peer-reviewed journal articles, magazine and newspaper articles. A three-part annex of supplementary information was reviewed together with four topical papers: The "First Nation Involvement in Remediation Efforts"; "Contamination Due to Insufficient Resources"; "Dangers to Food, Water, and Air Both on and off Reserve Lands"; and "Risk-Ranking," (which compares national and provincial classification methods for contaminated sites and the follow-up actions to be taken regarding remediation). The findings helped to inform the choice of which on-reserve sites to visit, the questions posed in key informant interviews, as well as the data and document review.

Document Review

A document review of the program was conducted, structured around identified evaluation issues and questions (see Appendix G, Documents Sources), that included the review of approximately 60 documents including, but not limited to:

• Reports produced by the Office of the Auditor General;
• INAC audits, evaluations, and special studies;
• FCSAP-related governmental documentation sites;
• INAC website and other related web pages;
• INAC Departmental Performance Reports and Reports on Plans and Priorities;
• Governmental Policy documents; and
• Provincial legislation and other related contaminated sites documents.

The document review helped to inform the evaluators' understanding of the governance structure of the CSOR, the broader FCSAP initiative and the roles and responsibilities of various parties involved in the program's delivery.

Database Review

The database review included the IEMS database (together with data on liability provided directly by the program), and a review of the publicly available Federal Contaminated Sites Inventory^{Footnote 26}, housed by the Treasury Board Secretariat. The database review was undertaken for several reasons, including viewing the CSOR Program's records of site investigations and cleanups, verifying the robust information collection and reporting nature of the IEMS, accessing information on reported sites, obtaining information on classified sites with confirmed (known) contamination, gathering information on where instances of site contamination mainly occur (e.g. on-reserve areas affected by site contamination issues that are the direct result of past practices such as fuel storage and associated spills and leaks), etc.

As a way of examining the program's relative progress across provinces in assessing and remediating contaminated sites, the database review included, among others, an analysis of level of classification of site by region. The IEMS database also yielded financial information that informed the effectiveness of the program's use of funding.

Site Visits

There were four site visits, which were primarily visual inspection of the sites and interviews with people involved in or linked to the sites (e.g., community representatives and project managers). Sites broadly illustrative of the CSOR were chosen among Manitoba and Ontario First Nation communities assessed as having high priority contaminated sites or were representative of various stages of assessment and remediation. The site visits also included walk-throughs, to identify areas of potential environmental concern to on-reserve communities. These were used to verify the information gathered during the literature review. Planned visits to communities in British Columbia could not be undertaken as the communities were unavailable. It was hoped that the opportunity provided by the evaluation would result in input from British Columbia reserve communities.

Key Informant Interviews/Focus Group

To increase understanding of the CSOR program and its delivery, 17 interviews were conducted in person or by telephone with individuals with direct experience and knowledge of the management of contaminated sites on-reserve. Those interviewed included INAC Headquarters (four CSOR) and regional staff. Interviews during site visits included First Nation community members and contractors. Focus group sessions were held (included eight regional and Health Canada staff).

2.2.1 Strengths and Limitations

The evaluation did not benefit from the presence of a working group since the program was short-staffed, managing competing priorities.

Barriers to effective implementation of the evaluation included: the absence of subject-matter experts with whom evaluators could have worked to achieve the evaluation's goals and objectives; staff turnover and lack of key member input into the evaluation plan; dependence on a variety of individuals during both the planning phase and the course of the evaluation. Nonetheless, the evaluation did benefit from recent audits and evaluations and, with FCSAP Phase III beginning in 2016, from current work on Phase III, including supplementary information from the research being undertaken for Phase III purposes. 

The CSOR Program's Alignment with Federal Priorities

As shown above (e.g., FCSAP), the CSOR is explicitly linked to the federal government's roles and responsibilities with respect to environmental management such as contaminated sites. In addition to FCSAP's original funding of $3.5 billion over 15 years announced in Budget 2004, the federal government has renewed support for the action plan through two additional funding endowments: the 2009-2010 Canada Economic Action Plan; and Budget 2011. In the case of Budget 2011, the federal government outlined its long-standing support of contaminated sites remediation stating that "The Government has developed a long-term Federal Contaminated Sites Action Plan to systematically assess, remediate and monitor sites for which the Government is responsible." The 2013 Speech from the Throne also addressed the third expected result of the CSOR, stating:

• "Canadian families expect safe and healthy communities […] and enjoy a clean and healthy environment"
• "our Government will continue to work in partnership with Aboriginal peoples to create healthy, prosperous, self-sufficient communities" (Government of Canada, 2013b).

FCSAP Phase III is set to begin in 2016, a further indication that federal contaminated sites constitute a serious priority for the federal government. This is, moreover, reinforced by the fact that the CSOR represents a significant portion of total liabilities, works closely with the FCSAP partner departments and is clearly aligned with federal priorities.

Graph 2 below highlights the significance of FCSAP in terms of alignment with federal priorities. As depicted by the graph, the CSOR, in most years, and especially in recent years, receives a higher proportion of FCSAP expenditures than its relative proportion of liabilities. This suggests that the CSOR's liabilities are a high priority for the federal government and are consequently well supported by FCSAP funds. It should be noted that, based on the literature review and key informant interviews, as well as the nature of the assessment, remediation and monitoring services procurement on-reserve, on-reserve contaminated sites may be socially more complex and in harder to remediate areas, thus justifying a need for more funding than sites that fall under other departments' jurisdiction. NOTE: the higher proportion does not mean that the CSOR receives sufficient funds.

Source of "Percent of Total FCSAP Spending" – Federal Contaminated Sites Inventory
Source of "Percent of Total Liabilities" – Appendix C, Table 1: Breakdown of Environmental Liabilities by Custodian Department

Text alternative for Graph 2: INAC's South of 60º Contaminated Sites On-Reserve in Relation to Total Liabilities and Federal Contaminated Sites Action Plan Spending

Line Graph 2 highlights the significance of FCSAP in terms of alignment with federal priorities. It shows INAC's CSOR in relation to total liabilities and FCSAP spending. The proportion of FCSAP expenditures is indicated along the vertical y axis for the years indicated on the horizontal x axis. The years are from 2007-08 to 2012-13 and the 2 lines indicate the percent of total FCSAP spending (line with intermittent small squares) and the percentage of total liabilities (line with intermittent large squares). In 2007-08 to 2008-09, liabilities rose from 0.04 to 0.1, then started dropping, down to just above 0.04 in 2012-13. In 2007-08 to 2008-09, expenditure rose from 0.06 and peaked 0.15 in 2010-11, then started dropping, down to 0.1 in 2011-12, then rose again to 0.12 in 2012-13.

In addition to FCSAP, one example of how the CSOR aligns with the federal government's priority regarding contaminated sites can be found in Chapter 3 of the Commissioner of the Environment and Sustainable Development's 2012 Spring Report. The report reviewed the status of federal contaminated sites, including sites on First Nations reserves. According to the report, contaminated sites do not only affect soil, water, and air integrity, but also have the potential to and may jeopardize the health of people who live or work near these sites. Furthermore, the report indicated that contaminated sites may harm natural flora and fauna. Thus, without remediation, the sites' negative effects could continue indefinitely since some of the contaminants, ranging from toxic and hazardous substances such as petroleum products to heavy metals and radioactive materials, are considered to be extremely persistent. Key informants and site visits to reserves confirmed these concerns.

The CSOR Program's Alignment with Departmental Priorities

It is important to note in relation to Aboriginal community priorities that there are 615 First Nations and over 80 percent have fewer than 1,000 people living on-reserve. Moreover, First Nations, particularly those living on-reserve, have higher unemployment rates and lower average incomes in comparison to other Canadians. Thus, with various First Nations viewing the development of reserve lands as either one of, if not the most viable means of generating wealth for their communities (e.g., developing natural resources or leasing  parcels of land to others, activities that spur economic development), contaminated sites do not augur well for such development. Furthermore, as reserve site visits showed, First Nation communities include homes, schools, churches, administration buildings, all of which are built on-reserve land and foster social development. The presence of contaminated sites, therefore, adversely affects communities.

Sustainable development (economic or otherwise) of reserve lands depends on First Nations' access to and control over their land and natural resources, including a clean and healthy environment. Thus, contaminated sites impact reserves' interests and remain a major barrier to economic and community development. In some areas, for example where development is expected and when the "shovel hits the ground," sites have to be abandoned because of the discovery of contamination. This, in turn, results in years of delay in developing the resource. Responsible development and use of natural resources has the potential to promote job creation and attract skilled resources and also to succeed, not only in the Canadian but also in the global economy. Contaminated sites deter progress, as case studies undertaken for this evaluation have shown and even economic development is adversely impacted if on-reserve land must be taken out of productive use.

Addressing contaminated sites on-reserve lands is expected to help in the attainment of development goals while also supporting government priorities related to Aboriginal training and employment. Reserve site visits showed that contaminated site management projects offer Aboriginals benefits in terms of work opportunities, skill development and knowledge-based career potential.

Where the reserve community lacks the capacity, the opportunity and the help, as well as the wherewithal, to develop and use its lands and resources sustainably for the community's benefit, then that particular First Nation reserve's community is deprived of an improved quality of life. In turn, the community is unlikely to approach the standard of health and wellbeing that other Canadian communities enjoy.

Departmental Priorities: Northern Contaminants Program - the Yukon, Northwest Territories and Nunavut

While this evaluation excluded the Northern Contaminants Program (which includes the Yukon, Northwest Territories and Nunavut), a brief comparison of the North and South of 60º programs may be useful in the context of the departmental priorities.

First, the nature or source of contamination sites in the South of 60º is mainly fuel storage and handling (67 percent), solid waste/landfills (20 percent) and other (13 percent) whereas in the North of 60º, past private sector resource extraction and military activity account for the contamination problem.

Second, in the South of 60º, contaminated sites are on-reserve, numerous, much smaller in size, tend to be within or in the close vicinity of communities, represent smaller financial liabilities, and new sites are expected to discovered anytime. Contaminated sites under Northern Contaminants Program management are fewer, large-sized, have a tendency to be far removed or are not in close proximity (i.e. in remote locations, away from any settlements), do not expect a significant increase in the number of new sites, and costly in terms of remediation efforts (Northern Contaminants Program sites are usually abandoned northern mine sites that will extend beyond FCSAP (2020).

The relative success of the North of 60º Northern Contaminants Program compared to the CSOR was identified as an issue by previous audits and program evaluations. A specific issue arose from the 2008 evaluation of both the North and South components of INAC's contaminated sites program, namely, a lack of dedicated funding on an annual basis made it difficult for both halves of the program to plan and prepare their site assessments and remediation processes accordingly. This was especially challenging with respect to the CSOR because funding is passed along to communities, adding an additional step between the funding source and the final contracting of services. In response to a recommendation made by the 2008 evaluation, the program has allocated some dedicated A-base funding for the program.

The evaluation also found that the placement of both the North and South contaminated site programs within the Department highlights how these two sub-programs are prioritized. For example, the fact that the CSOR is situated with the Lands and Economic Development Sector provides some indication about the role that the CSOR plays in ensuring that reserve land, aside from being healthy to live on or near, is effectively available for any economic development opportunities the community may want to pursue with it. Economic development for First Nations is a fundamental departmental priority, thus, the placement of the CSOR with the Lands and Economic Development Sector indicates the importance of assessing and remediating contaminated sites in order to create a healthy economic development environment.

Based on interviews conducted, however, some challenges regarding support (number of staff and funds) for the program exist. The evaluation found significant staff support for the CSOR, but the variability of program staff and resource availability for program delivery differ considerably across regions. Interviews with regional staff revealed that a number of environmental officers are handling both their CSOR responsibilities and other program duties as well. Accordingly, staff dedicated to CSOR in some instances could be as few as one individual. Regional staff also noted that since FCSAP funding is unavailable for new sites or sites that are not high-risk, resources were often under pressure in attempting to deal with non-FCSAP sites. As shown in Graph 1 (number of sites by classification across provinces), most regions are dealing with a number of class 2 and 3 and unclassified sites. There was some expectation among interviewees that possible changes in Phase III of the FCSAP program might address some of these resource needs. Despite a lack of financial resources, the regions noted, almost unanimously, that they felt well supported by Headquarters when funding was needed, identifying the national CSOR coordinator as particularly strong in helping to secure resources.

In summary, although distinctions exist between the contaminated sites that are North and South of 60º, with most (93 percent) of INAC's liabilities for contaminated sites found North of 60º, lessons learned from the latter can be applied to the South of 60º program. For instance, by adapting or using the contribution to scientific knowledge and expertise developed as part of the northern remediation efforts. See Section 6, "Other Findings/Observations" for more on this topic.

The CSOR Program's Alignment with Community Priorities

As noted, communities prioritize contaminated site clean-up differently, depending on other pressing issues they may be dealing with. The sheer numbers of suspected contaminated sites communities have identified and that are yet to be addressed points to the concern that on-reserve communities have regarding contamination.

Program staff mentioned that new sites are constantly being added to the inventory of suspected sites and during site visits, for example, the evaluation learnt first-hand of new sites that have surfaced in one reserve. The evaluation also learned from key informant interviews that communities that notify the Department of a potential contaminated site are interested in assessment and remediation. Given communities' hands-on role(s), often in order for the remediation process to begin, reserves which are preoccupied with other outstanding issues are sometimes unable to prepare a contaminated site for it to be "shovel-ready" when funding is allocated.

The promotion as well as the support of Aboriginal economic development, the use and management of reserve lands, including, in particular, the environmental management and the remediation of contaminated sites, are all federal, departmental and community priorities. They will continue to be central to virtually all forms of Aboriginal economic development. The Auditor General's 2009 Fall Report noted that reserve lands are central to First Nations peoples' history, cultural identity and day-to-day activities. As the evaluation found, this is a key reason why the assessment and remediation of contaminated sites in the South of 60º has a direct impact on the quality of life of First Nations. It is crucial to the economic benefits of the affected reserve and surrounding communities (as it allows and encourages the future development of lands that would otherwise be unusable) and, to the wider Canadian economy as a whole.

Observations/Consideration

As the evaluation has found, there are some risks that need to be addressed, one of them being the lack of control over the results of others' activities, which is also related to Third Party risk. For example, by allowing activities which continue to cause small scale pollution on-reserve, there is a risk that INAC may be held responsible for contamination as a result of activities which INAC does not perform and/or manage. Another key observation is that if the particular on-reserve community does not exercise diligence in prevention and monitoring, INAC may be liable for the contamination that may result. Thus, with INAC as a whole, there are difficulties in monitoring all activities on-reserve, yet INAC could be responsible, or for health and safety reasons, assume responsibility, for remediation.

In addition, due to inadequate policy and regulatory framework (which implies jurisdictional, unenforced/application of existing regulatory issues), there is a risk that INAC may be held accountable for contamination on-reserves. Thus there is a need for an understanding by policy makers and regulators about who is accountable for contamination prevention. If regulations are not adequately enforced, it is possible that the resulting contamination could fall under INAC's responsibility.

Building on the User's Guide on Determining INAC and Third Party Contaminated Site Responsibility, the CSOR may benefit from a strengthened liability regime. This could result in the availability of some needed funds to help pay for clean-up costs, while ensuring that on-reserve communities are protected in the event of pollution.

5.3 Unintended Outcomes

Though contaminated sites are considered injurious to both human and environmental health, the need to remediate and restore a contaminated site to (or close to) its original state has resulted in both positive and negative unintended impacts as a result of the program. The evaluation found that the CSOR has demonstrated a viable approach to supporting community initiatives aimed at improving capacity development. This has numerous long-term benefits, for example, an opportunity to train local people to work on site remediation projects introduces knowledge and the acquisition of transferable skills to community members with an interest in the site remediation project, as evidenced in case studies undertaken in northern Manitoba. Such knowledge and skills are transferable and can be applied to other types of projects, such as resource development.

As noted by interviewees, though an outcome that was intended, it bears mentioning the fact that there was indirect impact on-reserve revenue (revitalization contributed to development of the area in question). The construction work undertaken brought employment for community members, who were employed in various areas and skill levels. Some of the jobs created were short term, for example, the opportunity to rent equipment, while others were longer term in nature, such as training and certification to equip participants with knowledge and experience.

In this sense, the knowledge that was gained, such as the capacity to engage in community consultations and other remediation activities as a result of the CSOR has contributed to community members' capacity to participate more effectively in consultation with non-Aboriginals and in regulatory review of other development projects. This is combined with an unexpected increase in interest and awareness among communities of the need for appropriate fuel handling and tank management procedures, an activity that was previously considered normal (i.e., no one paid attention to the negative consequences of this practice). This is significant insofar as the CSOR's initial objective was to provide support and to remove real and perceived threats to health and safety, to people and to the environment.

Communities are experiencing other unexpected benefits from remediation and redevelopment, many of which are environmental, social and economic in nature. For example, the removal/ tempering and renovation of abandoned and derelict buildings, which in turn has led to a decrease in vandalism, trespassing, risk of injury and so forth as shown by site visits to on-reserve communities in northern Manitoba. Further, a newly rediscovered contaminated site is pointing to the need to preserve a historical landmark such as a cemetery.

In terms of communications, interviewees acknowledged that the ongoing remediation projects have created opportunities for CSOR staff (i.e. INAC) and communities to either develop or forge stronger working relationships with each other. Thus, an indirect impact, though not measured, is an improvement in the Department's reputation, which interviewees based on visible, tangible evidence of ongoing clean-up activities as well as environmental improvements.

6.1 Program Efficiency

One of the major challenges is evaluating the complex interplay between operational multi-unit^{Footnote 40} programs, and to assess their unique contributions to that particular program's success. One of the objectives of this evaluation was to examine the overall impact of the CSOR, which has several components, a pool of interrelated projects with various interrelated departments and multiple partners with respect to its funding and delivery. As a result, this can become difficult as each unit requires a separate analysis in a way that shows its overall impact and performance relative to other units.

As the previous as well as the following show, measuring a multi-unit program such as the CSOR can result in assessments that may be complex and/or tricky to interpret. For instance, the evaluation of the FCSAP offers a report where it assesses a unit, the FCSAP, using different types of evidence, resulting in information that can be difficult to compare with the CSOR simply because other departments have a role to play in the delivery of the FCSAP. These departments, which include INAC, have different levels of resources, thus, their inputs and outputs vary.

Measuring the CSOR's efficiency and economy prove to be difficult as its priorities are at times influenced by internal and external factors, which lead to project delays, sudden additional reporting requirements and other pressures. In addition, measuring in the social sector where outcomes are sometimes difficult to define, and often harder to measure is challenging. Despite these difficulties as well as the requirement to demonstrate that the CSOR is spending its (i.e. public funds) in an efficient and economical way, and that it is achieving a positive impact relative to its capacity, the evaluation has found, and as the data substantiates, that the CSOR has performed well in terms of the resource utilization in relation to the production of outputs and progress towards its expected outcomes.

Although the evaluation notes that the CSOR has had successes, consequently, its efficiency is difficult to measure when considering not only the previous paragraphs but also the main objective of the program - to manage contaminated sites. This involves program delivery through First Nations and presents both unique opportunities and challenges for completing projects. For instance, the CSOR priorities are at times influenced by project delays and other pressures and work does not always occur according to the risk ranking attributed to sites. Land claim agreements, transfer of lands through First Nations Land Management Initiative or Treaty Land Entitlements are typically assigned priority in the CSOR planning process. Moreover, planning is based on winter road accessibility for northern sites (site visits showed that often work must be done a year in advance to take account of winter road requirements), other ongoing projects in the particular reserve that may be of greater priority, and the capacity of the First Nation community to provide labour and/or equipment.

The CSOR's classification system and responsibility for a variety of sites at various levels of contamination is also an issue to consider. By way of illustration, even with the classification system, two sites ranked as "high-risk" may be difficult to compare because of other important factors. For instance, because the basis for the program is the delivery of funds to communities in order to cover the costs of assessment and remediation, while the expected outcomes are based on liability levels, the outcomes are, in a sense, alienated from what the program is able to do. Furthermore, unlike the North of 60º (Northern Contaminants Program) program, the CSOR does not generally play a role in choosing the resources for the assessment or remediation of sites. Overseeing the funding, while not being in a position to directly manage a project, suggests that outcomes may not necessarily be what were envisaged.

Instead of looking at expected outcomes to measure the CSOR's efficiency, other indicators may be more useful. For example, one measure of program efficiency is the average length of time it takes to remediate a site that has been assessed. The average for CSOR is 7.2 but this measure too should be viewed with caution for the reasons noted above.

Another measure of program efficiency is the manner by which funding is delivered to regions. As noted above, delays in funding are a major concern, particularly for regional program staff and for community members. This in turn leads again to the lack of dedicated funding, an issue that the program cannot be certain will be resolved. The lack of dedicated funds is particularly clear for multi-year projects where funding must be renewed annually. As the evaluation found, the Canada Economic Action Plan^{Footnote 41} added a sudden influx of money to FCSAP funding, which led to certain challenges. The Canada Economic Action Plan Program, for example, permitted little cash management flexibility due to the very short implementation time frame. Having more flexibility in cash management between fiscal years would have allowed the re-profiling of funds for a project that was delayed due to unforeseen circumstances (e.g., inclement weather). Interestingly, the sudden addition of funds was less problematic for the program than the additional reporting requirements that came with it, considering the capacity challenges faced by some communities. It should be noted that the addition of extra funding did not prove to be a problem for CSOR, although it was for other custodian departments.

Procurement challenges are linked to the sudden infusion of funds. While the extra money may have allowed for projects to be undertaken and given First Nations an opportunity to acquire more capacity, interviewees noted that communities with capacity constraints or a lack of expertise can and do find the procurement process challenging. In its own way, this detracts from efficient use of resources.

The literature review showed that public procurement is an important function of government for a variety of reasons, including, for example, the fact that the sheer magnitude of procurement outlays has a great impact on the economy and needs to be well managed. Thus, efficiently handling procurement outlays, which touches on First Nations interests, has been a policy and management concern as well as a challenge for public procurement practitioners. Added to this is the pressure that arises when procurement is, to some extent, being used as an important tool for achieving economic, social and other objectives for First Nations while ensuring that closer and greater scrutiny by taxpayers and competing vendors are handled professionally and well.

Procurement challenges facing the CSOR Program have included, for example, competition for consultants due to demand from the commodity/resource sector and the remote location of sites. These challenges were also mentioned during site visits. The impact was on the pace and cost of projects and managing procurement requirements normally include financial and technical risks, including quality, timeliness, cost (which is more than just the price), minimizing business while maximizing competition, and maintaining integrity. In spite of the lack of CSOR resources, it was noted that program staff offer help with the request for proposal processes and, where applicable, collaborate with expert departments such as Environment Canada, Health Canada and Fisheries and Oceans Canada.

In order to address issues raised by previous evaluations and audits, the program has developed a suite of policies and programs to enhance performance. Among these are two guidelines on cost-estimating for contaminated sites and remediation liability accounting. Outlining the principles used to determine final costs and liability figures helps ensure that, at the least, the various approaches used are consistent across regions and staff turnover. Liability accounting guides have also been developed at the department level to maintain consistency, not only across the CSOR but the North of 60º program as well. Consistency is important for program efficiency because it strengthens the process by which funding is delivered to the regions.

Document review indicates that the CSOR staff (at both the regional and Headquarters levels), administer the program and work with on-reserve communities to manage and implement remediation projects. In partnership, INAC and the particular Band Council concerned choose the consultants and engineering firms, which will implement the remediation plan, thereby allowing for project ownership and potential economic benefit for the community. Throughout a project's life site visits and interviews show that, where possible, First Nations members are trained on heavy equipment use, soil sampling, project management and construction techniques and so forth, all skills that are transferable and can be applied to many other types of projects. This allows involved First Nations to be responsible for choosing the firms that ultimately come into their communities, which makes sense as the former have both a better understanding and historical knowledge of their territory. Further, it demonstrates an efficient use of resources.

Finally, joint program delivery, part of the contaminated sites work, also builds partnerships and relationships on-reserve lands. This, in turn, encourages socio-economic benefits (e.g., training and capacity building, support of future economic development initiatives), given that the average remediation project lasts four years from start to finish. An example from the Toquaht Nation (British Columbia) of a benefit from a remediation project is the increased awareness of environmental issues among the on-reserve community. This has resulted in the development of an Environmental Management Plan, which is expected to guide all development, operations and maintenance programs on Toquaht lands. The plan is assessed on an annual basis to ensure it remains relevant and up-to-date, preventing harm to community lands and the marine environment.

Although the Toguaht Nation was not one of the sites visited for the evaluation, it provides an example of an intangible benefit whereby the community was able to use several community members and local companies for a project related to a bridge replacement. A total of 4,830.28 tonnes of soil was excavated, temporarily stockpiled, sampled and sorted for removal based on detected concentrations of contaminants. This practice also prevailed in other communities visited such as Garden Hill and Little Grand Rapids in northern Manitoba. The perception is that where there is sound environmental management and protection, there is assurance that lands are available on what could be considered a very limited reserve land base for First Nations to pursue opportunities in business development, residential expansion and cultural/traditional pursuits for their members. The demonstration of an efficient use of resources is evident here.

6.2 Program Economy

As mentioned earlier, the costs of inputs used for each of the CSOR's activities is subject to various factors, though funds available and the cost targets (i.e. budgeted costs) set by the program shows that the program attained success in its endeavours. Further, the differences between planned and actual spending for the CSOR's outputs provides evidence that in spite of its challenges, CSOR unit costs compared with other similar programs, example the FCSAP, has been encouraging.

As the evaluation found, though the CSOR's resources are not reasonable in light of intended outcomes, it is undertaking specific activities and delivering products at the lowest possible cost. On the other hand, given the number of contaminated sites at hand, in addition to emerging ones, the evaluation notes that not all the risks and financial liabilities associated with the CSOR's contaminated sites will be addressed by the current program resources.

The evaluation found that various factors at both the CSOR and project site levels either contribute to or negatively impact the program's efficiency. Thus, the cost-effectiveness and economy of the CSOR, for instance, have proven to be difficult to measure for the following reasons: (1) sites are constantly being added, thus, creating the need to be assessed (as a result, which makes it difficult for the program to reduce the number of its sites); (2) the fluctuating level of funding makes it difficult for regions to plan for funding (this likely results in less cost-effective decisions that have to be made at the last minute; and (3) the low human resource level. Key informant interviews, site visits, and document review showed that the CSOR has difficulty attracting and retaining staff to work on contaminated sites. Human resource gaps and problems in retaining staff are especially acute in the regions. With communities often having difficulties getting projects "shovel-ready" ahead of time, funds coming in late may simply be lapsed.

It should be noted that despite not having a strong base of dedicated funds, including insufficient and late receipt of funds, the program has seen decreases in the number of high-risk sites, the number of sites to be classified and the level of liabilities. This is likely due to the strength of the program staff at Headquarters who are able to procure the Department's share of the cost necessary to complement FCSAP funding, even if the accessible funds are not enough. However, the constant addition of new sites makes measuring challenging. As a result, if funding were in place in a more dedicated manner, program economy could be measured much more accurately. Similarly, by being somewhat removed from the activities related to the procurement of assessment and remediation resources, it is inappropriate to evaluate the CSOR on the relative economy of the resources that are ultimately chosen. Nevertheless, the CSOR's efficiency and economy are perceived to be encouraging when such factors as integrated management, coordination and shared tools and resources are considered as contributing to economy and efficiency.

First Nations communities on-reserve may have specific needs (for example, the remoteness of communities), which in turn necessitates specialized services. Aboriginal firms vying for work on projects may also have specialized skills related to land work on-reserve, which may lead to a different price level. One concern heard through site visits was that some communities considered that the assessment and remediation firms monopolize an area. Thus, for example, in a difficult to reach area, an established firm may have an advantage by already having in place and readily available difficult-to-transport equipment, allowing it to increase prices.

As the evaluation observes, such economic factors are external and impacts the CSOR's performance. As site visits demonstrated, there is competition for specialized environmental consultants who are often in high demand. Ironically, this increases business opportunities in the private sector (new commercial/industrial employment). Yet as a result, this adversely affects the cost and timely completion of projects while adding to the geographic and climatic challenges, which impinges on projects in communities, especially remote on-reserve communities. Correcting this sort of market inequality may be outside the program's mandate, but as regional environment officers already play an important advisory role in the Request for Proposals process for services, this could be an issue to which regional staff could pay particular attention.

Taking these afore-mentioned factors into account, including the need for dedicated funds and specific communities' needs, it is apparent to the evaluation that some of these hindrances to economy and efficiency, as identified by key informants at both the program and project levels, in some cases relate to the federal Government of Canada's processes, which are beyond the control of the program (e.g., rigorous and constraining processes to manage program funds, reporting burden). However, overall, in particular from the remediation standpoint, progress is being made by the program with respect to the CSOR's stated objectives, including improvement of health and safety of First Nations on-reserves. For more on economy, see also the finding on integrated management approach (below) that is taken by the CSOR.

By "best practice," the evaluation is referring to an approach taken by the CSOR and which, through the program's own experiences, has proven to reliably lead to a desired result. The CSOR has a large inventory of high-priority, FCSAP-eligible projects (Class 1) and high-priority ineligible projects, which includes Class 1 as well as Classes 2 and 3. The vast majority of these sites are a risk to human health and safety because of their proximity to residential areas. Furthermore, and which complicates the assessment and remediation processes, several are in isolated communities where equipment and skilled labour mobilization add significant costs to projects. As a result, these lend themselves to the need for an integrated management approach.

The CSOR developed an integrated management approach, which focuses on continuous improvement and leads to sustainable land use and operations by communities. The approach engages the on-reserve community through communications, housing replacement improvement, community land use plan, fuel and solid/hazard waste management. It also involves legal surveys and property records, environmental education and skills training, all of which are linked to the Treasury Board Secretariat's Contaminated Sites Management Approach (the Ten Step Process).

In response to a 2013 Audit of Environmental and Contaminated Sites Management (South of 60), INAC developed a departmental Environmental Management System, which coordinates environmental management within the Department.^{Footnote 42} As part of the work in developing the EMS, research and assessment on existing data, provincial legislation and regulations, federal regulations, compliance and monitoring was completed with regions. Furthermore, a Gap Analysis was conducted, which highlighted the departmental needs to address all its environmental aspects. A part of this integrated management approach included prioritizing efforts on the development of a waste management strategy on-reserve.

Finally, as a way of minimizing costs, the CSOR Program's integrated management approach includes coupling the remediation of FCSAP-eligible projects and non-FCSAP eligible sites that are in the vicinity, while addressing infrastructure renewal and improvement investments. Program staff consider that this is a cost-effective option for Canada, as these critical investments are made to take advantage of occasions where heavy equipment and skilled labour have been mobilized for use in other projects. While this approach requires considerable investment from INAC in addition to the cost-share amount, it results in reduction of Crown liability and the possible prevention of new contamination from occurring.

Funds

While funding has already been discussed elsewhere, the repetition here is intended to juxtapose the two programs in order to highlight the differences while emphasizing the need that has been identified in this respect. The Northern Contaminants Program employs Grants and Contributions to engage with northern communities or for consultation and engagement purposes, which is done through PWGSC^{Footnote 43}. This makes Public Works the owner of the program and also gives it control over the project. The CSOR, on the other hand, transfers its funds to third parties. Such a funding arrangement could distance the departmental expertise from the issue being addressed as the third parties ultimately end up doing the work. In a site visit focus group session in the North, the evaluation learned that, for example, consultants seemed to be taking on INAC's role. After the initial assessment is done, a third party is then paid to do a review (work that has to/can be done by INAC), which also requires more work leading to another contract because of limited resources to undertake the review.

The nature of the northern program's governance structure, where both funding and planning are centralized at Headquarters, strengthens its approach. For example, a Director General who is involved in the Northern Contaminants Program participates at the management level meetings and is therefore able to facilitate program operation. The CSOR appears to take a coordinating approach involving the various parts of the Department relevant to the program. The Northern Contaminants Program has "one voice, one strategy" as opposed to the CSOR which "doesn't have a profile and a strategy."

While use of PWGSC may appear to be a good practice for the CSOR to adopt, it should be noted that one reason why the Northern Contaminants Program uses the PWGSC is because its contaminated sites are very large. According to interviewees, Northern Contaminants Program sites are at least $2 million and contracting with the PWGSC is very expensive^{Footnote 44}; also, the Project Management is internalized by PWGSC; PWGSC no longer provides management services. Moreover, even if the CSOR were to use PWGSC, each regional office is different and any decision should be made on a site-by-site basis.

An issue related to "surplus money from the North being sent over to South" was raised by interviewees. According to key informants, surplus funds are sent from the Northern Contaminants Program to the CSOR and some $7 million has been transferred during the past five years.

Even though these distinctions exist between the contaminated sites that are North and South of 60º, with most (93 percent) of INAC's liabilities for contaminated sites found North of 60º, lessons from both programs can be applied to each other. For example, by adapting or using the contribution to scientific knowledge and expertise developed as part of the northern remediation efforts, especially since the Northern Contaminants Program has contamination issues which are not dissimilar from those of the CSOR, including diesel tank-related problems. It bears repeating, however, that the Northern Contaminants Program's situation in terms of land management regime (not reserve and/or provincial lands) is different from the CSOR's.

As discussed above, the CSOR faces significant funding challenges that result in a myriad of difficulties; for example, having a plan and a more strategic approach, which will provide the program beneficiaries with a consistent way to deal with contaminated sites. However, interviewees raised the possibility that within INAC, program efficiency might be increased by relocating the CSOR to the Regional Operations sector, away from the Lands and Economic Directorate where it is currently housed, in light of the environmental nature of its work. This is based on the recognition that the CSOR is more of a regional operations program broadly delivered by the regions and should thus be closer to community infrastructure thereby offering potential efficiency gains.

Within INAC, one of Community Infrastructure's responsibilities is to ensure that on-reserve community infrastructure is well planned, effectively managed and comparable to what is enjoyed by Canadians living off-reserve. Through the First Nations Infrastructure Fund, it helps fund among other things, roads and bridges, solid waste management, planning and skills development.^{Footnote 45} In addition, on-reserve infrastructure is funded primarily through contributions made under the Capital Facilities Maintenance Program^{Footnote 46}, a major programming element within the Community Infrastructure programming activity. One of the Capital Facilities Maintenance Program's stated objective is to ensure that on-reserve activities are undertaken in an environmentally sound and sustainable manner.^{Footnote 47} Thus, to meet its objectives, the Capital Facilities Maintenance Program funding "assists First Nations to acquire, construct, operate and maintain needed infrastructure assets and facilities including:…environmental site cleanup / remediation."^{Footnote 48}

Program location within INAC is also related to the governance (i.e. implementation) challenges facing the CSOR Program, which is linked to human resource capacity challenges. At present, while the roles and responsibilities (Headquarters and regions) are responsive to an identified need, it remains that at the Headquarters staff level, the CSOR's National Program Coordinator is mainly responsible for the CSOR implementation work, including acquiring the funds required to meet CSOR needs. While it appears that the Community Infrastructure Branch may be better positioned with respect to human resource capacity, this is also an area which gives rise to an opportunity for the CSOR to coordinate its activities with the other complimentary programs within the Department in order to improve First Nation communities' awareness and response to fuel and waste management issues.

Finally, as this section has demonstrated, the Department has developed various programs and initiatives in order to respond to the challenges faced by Aboriginal communities in such areas as health, social well-being, sustainable communities and fostering economic prosperity. These programs and initiatives have a broad range of objectives and thus, represent an opportunity for the CSOR to facilitate the long-term cleanup of South of 60 sites. Thus, for example, communities and INAC would benefit if the CSOR, together with the other programs, specifically those complimentary programs offered by the Community Infrastructure Branch, could identify best practices with regard to their objectives/goals, structures, modes of operation, synergies across the programs, as well as funding levels. Ultimately, this would make cleaning up CSOR sites easier.

Assessment:

Remediation/risk management/long-term monitoring:

As noted in the descriptions of the steps, sites are typically closed at step 2, 4, 9, and 10. However, there may be exceptions where projects are closed at a different step.

Text alternative for Stages of Assessing and Remediating a Contaminated Site

This figure displays the progression of ten "steps." Starting with Step 1, each box containing the step number and the step description lead into the proceeding step boxes with an arrow. The order of the boxes is as follows: Step 1 Identify Suspect Site; Step 2 Historical Review; Step 3 Initial Testing Program; Step 4 Classify Site; Step 5 Detailed Testing Program; Step 6 ReClassify Site; Step 7 Develop Remediation/Risk Management Strategy; Step 8 Implement Remediation/Risk Management Strategy; Step 9 Confirmatory Sampling and Final Report; Step 10 Long-Term Monitoring.