Evaluation of the First Nations Water and Wastewater Action Plan

1.2.1 Background and Description

Access to safe drinking water, the effective treatment of wastewater and source protection for First Nations and on First Nation lands is a Government of Canada priority. The FNWWAP is designed to improve the health and quality of life of people in First Nation communities by assisting First Nations in providing better water and wastewater services to their communities.

The FNWWAP was introduced in 2008 as a horizontal initiative with Health Canada as part of government commitments in the 2007 Speech from the Throne, and further reinforced in Budget 2008, Budget 2010 and Budget 2012 to support First Nations' access to safe drinking water. It is a successor to the joint First Nations Water Management Strategy (2003-2008) and the AANDC Plan of Action for Drinking Water (2006-2008). While the current evaluation covers the period up to the expiration of authorities in March 2012, the FNWWAP has since been renewed and current authorities expire March 31, 2014.

From 2008 to 2012, the Government of Canada invested nearly $556 million in the FNWWAP, and a total of nearly $1.4 billion on all water and wastewater activities.

1.2.2 Program Objectives and Expected Outcomes

According to its originating policy documents, the objective of the FNWWAP is to support First Nation communities on reserves in bringing their drinking water and wastewater services to a level and quality of service comparable to those enjoyed by other Canadians living in communities of similar size and location. The FNWWAP was approved from 2008 to 2010 and renewed for the period from 2010 to 2012 in order to:

• Continue investments in the capital construction, operation and maintenance of water and wastewater systems, as well as investments in operator oversight, including drinking water quality monitoring initiatives by Heath Canada's Drinking Water Program;
• Develop a legislative framework for First Nations aimed at protecting water quality; and
• Implement initiatives to enhance the efficiency of existing investments in water and wastewater infrastructure.

The stated outcomes of the FNWWAP were to ensure that:

• First Nation communities have an increased capacity to address potential water problems;
• Health risks associated with water quality and supply are minimized;
• All First Nation community water and wastewater facilities meet federal standards; and
• First Nation communities have increased confidence in their drinking water.

To meet these objectives, FNWWAP invested in five key activity areas: infrastructure investments, operations and maintenance, training, monitoring and awareness, and standards.

1.2.3 Program Management, Key Stakeholders and Beneficiaries

AANDC and Health Canada are the federal departments that are primarily responsible for funding the delivery of safe drinking water on reserves.

1.2.4  Program Resources

From 2008-09 to 2011-12, FNWWAP (investments to all departments and Grants and Contributions) invested $556 million to improve water and wastewater services on reserve. In total, AANDC spent $1.389 billion on water and sewage infrastructure from 2008-09 to 2011-12. FNWWAP funding represents 47.35 percent of AANDC's total investments. Health Canada spent $68.7 million on FNWWAP from 2008-09 to 2011-12. Original proposed expenditures specific to FNWWAP are detailed in Table 1 and actual reported expenditures for all water and wastewater activities reported by AANDC's Office of the Chief Financial Officer are detailed in Table 2. Expenditures provided from the financial system in Health Canada are provided in Table 3.

Governance

AANDC was the lead on this evaluation. A Memorandum of Understanding was established between AANDC and Health Canada to outline the conduct of the evaluation, underscoring AANDC's role as principally responsible for data collection and generating a final report. Health Canada was responsible for conducting additional analysis to inform some of the key evaluation questions, reporting on the achievement of outcomes relative to Health Canada's responsibilities and contributing to the final report. Health Canada-Public Health Agency of Canada's Evaluation Directorate and the Health Canada's First Nations and Inuit Health Branch's Performance Measurement Unit and program staff reviewed and provided input to the evaluation report.

To better ensure quality and transparency, the evaluation included an Advisory Committee, whose mandate was to provide insights on evaluation tools, findings and recommendations. It was composed of representatives from the Assembly of First Nations, AANDC and Health Canada. Environment Canada was not explicitly involved in this evaluation, as it is not specifically involved in the FNWWAP.

The Evaluation, Performance Measurement and Review Committee (EPMRC) acts as a review and advisory body for AANDC evaluations and only approves evaluations that have successfully fulfilled the requirements and needs of both AANDC and Treasury Board Secretariat. Health Canada's Executive Committee acts as the review and approval body for all Health Canada evaluations.

Relevance

Continued Need

• Is there a continued need forthe investment in water and wastewater initiatives?
• Can the FNWWAP be reasonably expected to reach its stated objectives?

Alignment with federal government priorities and departmental strategic outcomes

• To what extent is the FNWWAPconsistent with:
  • Federal government priorities; and
  • Departmental Strategic Outcomes.

Alignment with roles and responsibilities

• Are the current roles of the federal government and the roles of the departments involved appropriate?

Performance

Effectiveness

• Has the FNWWAP been achieving its intended outcomes? Have there been improvements to water and wastewater management and capacity?
• Have there been positive or negative unintended outcomes? If so, were any actions taken?

Efficiency

• How has the FNWWAP optimized its processes and the quantity/quality of services to achieve expected outcomes?
• How can the FNWWAP's efficiency be improved?

Economy

• Are there opportunities to achieve the intended results of the FNWWAP with fewer resources?

Design and Delivery

• Has the FNWWAP's design and the means at its disposal contributed to the achievement of the intended outcomes?
• Is the FNWWAP implemented as planned? If not, why?
• Are the governance structures and roles and responsibilities clearly understood?
• Are they effective in achieving the expected results?
• Could they be improved?
• Is a system in place to identify, collect and report on:
  • Risk?
  • Performance measurement data?
  • Is the performance measurement data collected used in decision making?
  • Is it working well? If not, could it be improved?

Other Issues

Lessons learned/Best Practices

• Did the FNWWAP take into account the lessons learned from the previous evaluation recommendations for both the First Nations Water Management System and the National Assessment at the design and delivery stages in order to maximize the success of the investment?
• Are there any lessons learned/best practices that could be used from the FNWWAP?

Alternatives

• Are there other means by which the program can achieve the same results more efficiently and/or economically?

2.3.1 Data sources

The evaluation's findings, conclusions and recommendations are based on the analysis and triangulation of five lines of evidence: document and file review, literature review, key informant interviews, surveys and case studies.

2.3.2 Considerations, Strengths and Limitations

4.1.1 Outcome 1: First Nation communities have an increased capacity to address potential water quality problems

Health Canada assessed progress toward this outcome by looking at the increases in the number of Community-based Drinking Water Quality Monitors (CBWMs); the level of community access to on-site water test kits; the frequency of water monitoring at tap; and the number of water and wastewater servicing projects reviews completed.

The analysis noted continuous increases in the number of CWBMs or Environmental Health Officers (where no CBWM is present). Access to water testing kits for bacteriologic parameters was also noted as a positive step as the proportion of communities with access to these kits increased from 92 percent to 100 percent since 2008.

In terms of water monitoring, Health Canada recommends weekly water quality monitoring of community water systems, and data analysis from 2007 to 2011 indicated that the percentage of communities meeting this target had increased from 44 percent to 61 percent. For routine chemicals, Health Canada recommends annual monitoring. Data analysis indicated that the proportion of communities meeting this target was static at 86 percent between 2007 and 2011. For bacteriological parameters, Health Canada recommends quarterly monitoring of semi-public water systems. Data analysis indicated that the percentage of communities meeting this target had decreased slightly from 69 percent to 62 percent over this time period; however, this was at least partially attributed to a change in methodology for data collection. The data further showed that with respect to cisterns (with guidance for monitoring released in 2012), 52 percent of communities were meeting the recommended annual monitoring.

In 2010, Health Canada introduced a Policy on Individual Wells^{Footnote 17} offering public awareness materials to prevent contamination, including educational materials for visually inspecting and maintaining wells; and on-request bacteriological sampling and testing services. The analysis in 2011-12 showed that 20 percent of individual wells were monitored for bacteriological parameters. Although this is an on-request service and not part of the performance measurement of the FNWWAP, monitoring wells plays a role in drinking water safety on reserve.

Health Canada has developed a variety of public awareness materials, guidance documents and procedures since 2008 to promote drinking water safety. Public opinion research indicates that receiving information about drinking water monitoring procedures and results can provide reassurance about the safety of tap water.

Between 2007-08 and 2011-12, all 240 water and wastewater servicing projects proposals received from AANDC were reviewed by Health Canada from a public health perspective. The overall objective of the review was to protect public health by ensuring that proposed water and wastewater systems include adequate protection measures that will minimize potential public health risks by addressing potential corrections to proposed designs. This approach is intended to help ensure that new or upgraded facilities operate as per standards and requirements.

Interview respondents all agreed that capacity to manage and monitor water and wastewater infrastructure has improved markedly since the introduction of FNWWAP. It was also widely acknowledged that AANDC has become more proactive in supporting communities that have limited local capacity. As of 2012, according to the most recent Annual Performance Inspections, 60 percent of water system operators and 54 percent of wastewater system operators had been certified to the level of their system – an increase of nine percent and 12 percent, respectively. However, only approximately one third of survey respondents felt that there were sufficient numbers of trained water and wastewater system operators in their communities, relative to their level of need. Most respondents did indicate, however, that the operators they had were certified to an appropriate level. When asked to comment, survey respondents largely cited issues of limited numbers of operators; extreme difficulties with recruitment and retention; and significant staff turnover as their main capacity barriers. The same one third of participants felt that there were sufficient numbers of individuals trained via the Community-based Drinking Water Quality Monitor Training Program or the Circuit Rider Training Program.

On Health Canada's side, CBWMs and Environmental Health Officers are the primary service providers with respect to drinking water quality monitoring at tap in First Nation communities. CBWMs are community members, funded by Health Canada, to sample and test drinking water for potential bacteriological contamination, while Environmental Health Officers train and assist CBWMs with their responsibilities. If a community does not have a CBWM, an Environmental Health Officer monitors the drinking water with permission from the community. Although most First Nation communities prefer to use the services of CBWMs for bacteriological monitoring, some prefer to only use the services of an Environmental Health Officer. In 2008-09 and 2011-12, all First Nation communities had access to either a CBWM or an Environmental Health Officer (or to both) to monitor drinking water quality at tap. Almost all CBWMs (96 percent in 2008-09 and 98 percent in 2011-12) involved in drinking water monitoring were trained. Although retention was not explicitly part of the program mandate, it was anticipated that the national training program developed by Health Canada would help increase the stability of the CBWM workforce, potentially by broadening appreciation of their importance in ensuring drinking water safety in their communities.

As well, Health Canada has made improvements to quality assurance/quality control requirements and monitoring procedures for safe drinking water in First Nations communities. The revised Quality Assurance Program for microbiological monitoring was developed by the National Drinking Water Quality Safety Quality Assurance Working Group. The revised Quality Assurance Program for microbiological monitoring includes a combination of quality control procedures and quality assessment activities. It states that, when off-reserve laboratories are used for analysis of microbiological samples, these must be accredited. For samples analyzed on-site, third party assessments of on-site water testing facility practices, including CBWM techniques are recommended every two years and the outcomes reviewed by the Environmental Health Officer. For quality control of on-site water testing facilities, a commercially-prepared positive and negative control must be analysed a minimum of monthly (ideally in conjunction with each sampling set). The revised Quality Assurance Program for microbiological monitoring is being piloted.

Emergency response procedures have also been developed to provide a standard response to potential drinking water emergencies. In 2011, Health Canada released the Procedural Guidelines for Waterborne Disease Events in First Nations Communities South of 60°. These initiatives, once fully implemented, will support First Nation communities' capacity to address potential water quality problems.

Despite the recent approval of the Safe Drinking for First Nations Water Act^{Footnote 18} there is concern that an improvement in drinking water safety is not likely with the implementation of a regulatory regime where there is limited local capacity. For example, as suggested by the 2011 Expert Panel on Safe Drinking Water for First Nations^{Footnote 19}, a serious investment in training and operational support is what is required to provide a safe drinking water supply, more so than complex equipment. Further, monitoring the compliance of maximum acceptable concentrations of pollutants may sidestep the actual causes of unsafe drinking water. This point is further reinforced by research on source water protection,^{Footnote 20} suggesting that complex treatment technologies and facilities would not eliminate the risks of unskilled or untrained monitors.

While according to interviewees and many survey respondents, developing capacity is a cornerstone of the Circuit Rider Training Program, capacity may mean different things depending on a community's circumstances. For example, key-informants cited circumstances where local capacity could actually entail the means to reliably outsource water and wastewater projects and monitoring or operations and maintenance, particularly in circumstances where sufficient local capacity may not be feasible or realistic because of a lack of qualified or interested individuals. In many cases, communities simply have too few system operators and the existing operators are overworked. It was further emphasised by survey participants that their primary concern was a lack of backup operators and/or an inability to fill positions where operators vacate their positions or are otherwise unavailable.

4.1.2 Outcome 2: Reduction in health risks associated with water quality and supply

Results of the National Assessment have shown that of 807 water systems examined in 2011, 39 percent (314) were classified as having a high overall system management risk, 34 percent classified as medium system management risk, and 25 percent classified as low system management risk.

The 2011-12 Annual Performance Inspection results showed some improvements, with 28 percent classified as high; 36 percent classified as medium and 36 percent classified as low. More specifically, 29 percent of communities decreased their risk rating, while 14 percent increased and the remainder were unchanged. Consistent with observations in Section 5.1.1, operator qualifications and record-keeping account for 60 percent of the measured risk, underscoring the importance of having certified and trained operators. Additionally, the majority of high risk systems serve a small population. The perceptions of survey respondents generally align with the official data on risks; specifically with 24 percent suggesting that they have seen a reduction in the number of health risks since 2008, 62 percent saying they have seen no change, and 14 percent saying they have seen an increase. When asked about the number of drinking water advisories communities have had since 2008, about 45 percent said they have had between one and three; whereas nine percent say they have had more than ten, and 28 percent say they have had none. In terms of waterborne diseases and outbreaks, 12 percent say they have had between one and three, whereas three percent say they have had more than ten, and 84 percent say they have had none. However, despite these survey responses, Health Canada data indicated that there were no confirmed cases of waterborne disease, and no identification or confirmation was made by a laboratory of any outbreak of water borne disease from 2007-08 to 2011-12.

The National Assessment further observed that, while not necessarily high risk, 161 water systems in 116 First Nation communities were under Drinking Water Advisory (DWAs) as of February 2011. These DWAs may be impacting up to 18,900 people, or approximately 3.9 %of the total on-reserve population (cited as 484,321 in the National Roll-up. Further, 17 percent of water systems and 21 percent of wastewater systems are operating either at or beyond water treatment capacity. As reported by Health Canada, drinking water advisories have increased by about 35 percent, from 269 to 363 between 2007 and 2011. Drinking water advisories can be broken down between boil water advisories (from 253 to 355) and "do not consume"^{Footnote 21} advisories or orders (which decreased from 16 to 7). The number of drinking water advisories may be increasing because more water systems are being monitored more frequently. As such, the increase in drinking water advisories can be viewed as a success insofar as the ability to detect potential health threats (particularly considering the absence of confirmed waterborne illnesses). However, the increase in drinking water advisories can also be a concern given the high frequency of risks detected, which suggests infrastructure or maintenance problems are common.

Although drinking water advisories have increased, most are revoked on average within a year of being issued with short-term boil water advisories being revoked on average within three months. The most common reasons for issuing drinking water advisories are inadequate disinfection or disinfectant residuals, unacceptable microbiological quality, equipment malfunction during treatment or distribution. Water systems under drinking water advisories for equipment malfunction during treatment or distribution appear to increase the average duration of advisories likely because these reasons are more difficult to address and required more resources. As such, the number of long-term drinking water advisories is also increasing.

Case study participants indicated that boil water advisories are most often implemented because of freezing pipes in winter, flooding, unpredictable water sources (i.e., lakes and rivers), infrastructure malfunctioning or systems operating beyond capacity. Communities where new plants had been constructed indicated that the number of advisories had reduced subsequent to construction, suggesting that new infrastructure does play a role. Given the relative newness of these infrastructure projects, however, it is not possible to tell whether the improvements with respect to boil advisories was because of equipment that was simply new and thus, less apt to fail, or because the equipment was customized to the unique needs of the community (i.e., source water, population, climate and geography) and was being operated and maintained properly.

About 45 percent of survey respondents stated their water and wastewater infrastructure was good or very good, while 35 percent said it was satisfactory and 20 percent said it was poor or very poor. About 37 percent of respondents said that it has improved since 2008, whereas 46 percent said it has remained unchanged and 17 percent said it has worsened. Participants' ranking of whether or not the infrastructure has improved was highly dependent^{Footnote 22} on their ranking of whether or not the water quality has improved. Most comments among those saying the infrastructure has become worse related to issues with equipment and aging infrastructure. Similarly, those who indicated it has improved cited specific infrastructure projects and investments as contributing to improvements.

Importantly, the measures above do not adequately speak to water source quality and risk. As discussed in more detail in Section 4.2.2 in the sub-section on "comparability", the measures currently collected have yet to yield the kind of data necessary to analyse water quality and risk improvements beyond simply counting the frequency and duration of DWAs. It was noted in Part 2 of the Report of the Walkerton Inquiry^{Footnote 23} that the first barrier to the contamination of drinking water involves protecting the sources of drinking water. In that report it was recommended that the province of Ontario adopt a watershed-based planning process.^{Footnote 24}

4.1.3 Outcome 3: All First Nation community water and wastewater systems meet federal standards

Following the completion of the National Assessment, AANDC committed to identify concrete actions in the areas of capacity building and training, enforceable standards and protocols, and infrastructure investments. In 2011, AANDC followed up with inspections through the department's Annual Performance Inspections cycle, which assessed water and wastewater systems in order to determine their risk management levels and progress since the National Assessment. Over the course of the second two years of the FNWWAP, AANDC contributed $424.7 million in capital funding and $253.4 million in operations and maintenance funding for water and wastewater projects. Projects funded included the construction of new treatment facilities and water and wastewater storage facilities, the expansion of existing systems, and the servicing of lots for new home construction. Over the same period, 48 major water and wastewater infrastructure projects were completed. Additionally, new Circuit Rider Training Program guidelines were developed to help standardise the program and to support system operators in improving the management of operations and maintenance of their systems. The Annual Performance Inspection results (2011–2012) indicated that 60.1 percent of water treatment systems and 53.9 percent of wastewater systems are now managed by operators certified to the level of the system.

Having in place effective standards and technology was another recommendation of the Report of the Walkerton Inquiry, as well as adopting a government-wide drinking water policy and Act (for Ontario). Respecting First Nations, the Government of Canada has moved forward with the Safe Drinking Water for First Nations Act, which now allows the Governor in Council, upon the recommendation of the Minister of Aboriginal Affairs and Northern Development, to make regulations governing the provision of drinking water and the disposal of wastewater on First Nation lands (although the latter was already in place via the 2012 Water Systems Effluent Regulations^{Footnote 25}). This is the first step toward what could be considered "federal standards" in the sense that this is the first stage of developing a federal standard or compliance mechanism that can now be articulated clearly and enforced. With respect to water and wastewater systems specifically, the Act allows for the creation of regulations for the location, design, construction, modification, maintenance, operation and decommissioning of drinking water and wastewater systems, as well as the distribution of drinking water by truck.

While the outcome of meeting federal standards may seem reasonable, there is concern that the focus on the authority for legal enforcement, as opposed to operational competence and knowledge transfer, may be overlooking key limitations with the current system.^{Footnote 26} The biggest successes noted by survey, interview and case study participants related to major infrastructure upgrades, and the biggest limitations related to human resources and limited funding for Operations and Maintenance – there was no discussion on major limitations being the result of a lack of federal regulations. The current Act does not necessarily imply funding obligations, and there is no reason to assume that enforceable regulations will help enable communities to meet federal standards without adequate capacity and infrastructure investments. Interviewees expressed some anxiety that the Act may represent a regulatory framework for the federal government with significant financial obligations for First Nations, and those financial obligations may or may not be adequately supported by the federal government.

Ultimately, the investments in infrastructure and capacity as well as major capital investments are ongoing (currently with FNWWAP authorities in effect until 2014), and the Act allowing for federal standards only came into effect in June 2013. It is thus too soon to tell whether these initiatives will enable communities to meet federal standards as they have not yet been fully articulated, and there is concern that the creation of enforceable standards will not address the real issues facing First Nation water and wastewater systems. The regulations are intended, however, to clarify the standards to be met, and provide the means to measure whether or not they are actually being met within any given system.

4.1.4 Outcome 4: First Nation communities have increased confidence in their drinking water

Perceptions of water quality and safety are difficult to measure and do not necessarily equate to actual quality and safety; however, one can assume that if there are consistent indications of improvements in safety and quality, particularly to a point where it is consistent with other Canadian municipalities, then confidence will increase.

Among survey respondents, 62 percent generally agreed that the quality of their drinking water was comparable to that enjoyed by other Canadians in communities of similar size and isolation. About 15 percent were not sure, and 23 percent did not agree. Whether their community had a water treatment facility or not seemed to have no bearing on this estimate.^{Footnote 27},^{Footnote 28} Further, 71 percent stated that the quality of their drinking water was good or very good; while 16 percent stated it was satisfactory and 13 percent stated it was poor or very poor. Thirty-nine percent said that their water quality has improved while about seven percent said that it had worsened, with 54 percent suggesting there has been no change. When asked to rank what they felt would be their community's level of confidence in the drinking water, 70 percent generally agreed that their communities have confidence in their drinking water, while 14 percent were unsure, and 16 percent disagreed. Among those disagreeing, there was an equal mix of those viewing this lack of confidence as based on actual risks and those viewing it based on a lack of information and trust issues with the water supply (often due to past problems or frequent advisories).

Opinion polling conducted by EKOS Research Associates in 2007, 2009 and 2011^{Footnote 29} showed some modest improvements in confidence over time. The proportion of respondents ranking their water quality as "bad" decreased from 33 percent to 25 percent from 2007 to 2011, and those ranking it as "good" increased from 44 percent to 49 percent. Rankings of the safety of their tap water also increased steadily from 62 percent ranking it as somewhat or very safe in 2007, to 71 percent in 2011. Critically, however, there was a sharp divide between First Nation communities and other small communities, where 21 percent of First Nation respondents suggested their treatment procedures or facilities were outdated or unsafe, versus seven percent of non-First Nation respondents. The EKOS survey also indicated that 68 percent of First Nation respondents used bottled water, compared to 40 percent of non-First Nation respondents. Generally, results of the EKOS survey were consistent with those of the AANDC evaluation survey.

4.2.1 Program Design

As discussed in Section 5.1.1, First Nation communities are facing significant challenges acquiring, training and retaining certified operators. Interviewees, survey respondents and case study participants highlighted turnover due to more competitive wages being offered by non-First Nation communities as one of the central reasons, as well as being able to afford, acquire, train and retain backup operators - all cited as major limitations facing First Nation operators in minimising risk. Importantly, however, several case study and interview participants noted the need for flexibility and customisation of approach. For example, there may be instances where a community's size and ability to acquire the capacity for water and wastewater management would implicate the need for contracting out services as opposed to building local capacity, and it was said that AANDC practices should be flexible and support this approach where appropriate. The Report of the Walkerton Inquiry had made a similar recommendation regarding special cases, and specifically First Nations, in that variances from regulatory standards should be allowed, but only where the owner demonstrates that safety will not be compromised, and never for cost purposes alone.

The available evidence in this evaluation emphasises the need to match investments in capital facilities with investments in local recruiting and training for long-term success. There are significant risks in investing in complex and expensive equipment where it is well-known that there is a high unemployment rate or a low likelihood of being able to recruit operators. It has been said that it would be better to have limited infrastructure with a highly competent operator than the best infrastructure with an inadequate operator,^{Footnote 30} and there was sentiment among interview participants that the FNWWAP focuses more attention on the infrastructure projects without adequate attention to the needs for sufficient numbers of qualified operators.

Case study and survey participants noted that communities are limited in their ability to attract qualified staff and have them sufficiently certified, at least partially because of a lack of available individuals having completed high school, and thus, this is a key limitation when resourcing a community for water system operators. Case study participants also discussed the need for incentives to attract or retain operators, such as housing funding, and the funds to allow for a full-time backup operator.

Other issues included:

• Cash flow issues such as requiring payment from the band up-front for training;
• The need for better communication, particularly when policy changes are in the works; and
• Communicating and discussing the specific reasons for not approving proposals.

Additionally, whereas AANDC normally promotes the use of decentralised systems, there may be instances where centralised systems are more efficient, and it was recognised by key informants that AANDC could be more cognizant of and flexible toward water system needs. It was also noted that better communication was required around specifying such needs. For example, where certain types of systems are optimal, AANDC could improve its communications as to why, in order to build better trust and understanding among First Nation water system operators and managers.

The protocols for the management of water and wastewater systems are said to be useful when understood and applied, but difficult to enforce and there is evidence that they are not as well known as would be expected for a program of such high risk. Only 39 percent of system operators surveyed were generally or completely familiar with the "Protocol for the Centralised Drinking Water Systems in First Nation Communities." Seventeen percent were somewhat familiar, and 44 percent were not familiar with it. Only 31 percent were generally or very familiar with the "Protocol for Centralised Wastewater Systems in First Nation Communities", with 21 percent saying they were somewhat familiar, and 48 percent saying they were unfamiliar with it. Only 24 percent were familiar with the "Protocol for Decentralised Water and Wastewater Systems in First Nation Communities", with 20 percent saying they were somewhat familiar, and 55 percent saying they were unfamiliar.  Knowledge of each of these protocols was highly intercorrelated.

There is little recourse for not meeting standards or protocols, although it was widely acknowledged among interviewees that the Safe Drinking Water for First Nations Act is the first step to addressing this issue. It was emphasised that the success of water systems stemming from the development of clear regulations would depend on the Government of Canada matching capital investments to the degree of upgrades required to meet regulations; on First Nation readiness for devolution of authorities related to water and wastewater activities where applicable; and on a very clear articulation of responsibilities between Health Canada, AANDC and First Nations.

Finally, as discussed in Section 4.1, the Government of Canada's approach to water and wastewater likely needs a longer-term focus on operations and maintenance that moves beyond short-term funding influxes for infrastructure projects. Considering the amount of additional infrastructure projects that can be expected in the future, the limitations in having qualified operators, and the relatively slow pace of improvement that can be expected with such limitations, a longer-term strategy may be needed. Specifically, it is difficult for both the Government and First Nations to dedicate to long-term and sustainable approaches to safe water and wastewater management if funding is variable, short-term and unpredictable.

4.2.2 Performance Measurement

The performance measurement aspects of FNWWAP are held within the broader performance measurement strategy for the Capital Facilities and Maintenance Program. Issues specific to water and wastewater within this performance measurement strategy primarily relate to risk levels and communities meeting the standards outlined in each of the protocols.

While measures of reductions in health risks are covered by Health Canada activities and measures for system risk and capacity have been operationalised at AANDC, measures of increased confidence and comparability with other Canadian municipalities have not been articulated or operationalised. Additionally, while a key outcome is that systems meet federal standards, measurement has not been articulated or systematised, and there are questions regarding the utility of measuring against standards that are not part of any regulatory framework. Regarding the latter point, regulations stemming from the Safe Drinking Water for First Nations Act should poise AANDC, in consultation with Health Canada and First Nations, to develop clear performance measures against the standards to be set in these upcoming regulations.

Finally, as discussed below, AANDC and Health Canada do not manage their financial information with respect to the projects funded under FNWWAP in a way that would allow for an effective assessment of efficiencies.

Comparability

Drinking Water Advisories

Table 4 below compares the "current" (as of May 7, 2013) raw numbers of DWAs by region between on- and off-reserve communities. The challenge with using drinking water advisories as a proxy for comparability is three-fold: 1) there is a challenge in collecting the right data from "comparable" communities with respect to numbers and durations of advisories; 2) it is difficult to conceptualise DWAs as a measure of success or failure of a system, as in a sense it could be seen as both; and, 3) DWAs are preventive measures and there exist many triggers leading to the issuance of DWAs. Drinking water quality is one of the triggers but not the only one. An additional difficulty is that an appropriate comparison would require comparing the proportion of systems under a DWA, and thus, would require an accurate knowledge of the total number of water systems in each province, as well as the ability to compare systems via the population covered. This would entail comprehensive research. Another option is to benchmark performance by setting more specific targets with respect to water quality and infrastructure stability and sustainability, as well as risk.

Beyond DWAs, comparability as it is articulated in AANDC documents is largely subjective insofar as it relates to infrastructure and water quality. There are multiple dimensions of comparability – most notably the suitability of the systems to the community given its needs, population, growth and geography; the quality of water; the stability and sustainability of the system; and the ability to maintain and monitor the system. AANDC has not articulated what aspects should be comparable and has not operationalised measures of comparability. At a minimum, the system risk indicators could be compared with geographically similar non-First Nation communities of approximately the same size. Comparing the quality of the water, however, is much more complex as there are so many measures, indices and variability in reporting. Federally, while standards for drinking water quality are within the purview of Health Canada, there is no systematic federal approach to measuring water quality off reserve as this is managed by municipalities and largely subject to provincial guidelines. (Although, it should be noted that Health Canada monitors drinking water quality on reserve at a comparable level with provincial guidelines).

One way to approach comparability would be through direct comparisons between one community's water quality and another via periodic comparisons of pH, toxins, clarity, bacteria, and biological sampling. As discussed in Section 1.2.3, currently Health Canada works with First Nations to measure total coliforms and E-coli, free and total chlorine residuals, baseline and routine chemical concentrations, and disinfection by-products. The test results are stored in databases that can be extracted, listing individual testing results for each of the dissolved concentrations of various chemicals and elements, largely detailed by date and the concentration in volume per litre. Similar data are collected by municipalities.

Table 5: Number of Existing Drinking Water Advisories as of May 7, 2013, including Precautionary, Boil, and Do Not Consume Advisories

Region	First NationsFootnote 31	Municipal/Provincial	Comments regarding Municipal/Provincial findings
Atlantic	9	257 (does not include Prince Edward Island - PEI)	Nova Scotia – 35 Boil water advisories Newfoundland/Labrador – 222 Boil water advisories New Brunswick – no current Drinking Water Advisories PEI – no information available
Quebec	2	173	Boil water and do not consume advisories for municipal and non-municipal systems from 17 administrative regions.
Ontario	75	N/A	No list available.
Manitoba	2	124	Boil water, drinking water avoidance and water quality advisories for public, semi-public and private systems.
Saskatchewan	13	244	Includes Precautionary Drinking Water Advisories and Emergency Boil Water Advisories reported by Ministry of the Environment or the Health Region in relation to public and private systems.
Alberta	26	0	DWAs included as "Active Health Advisories". Alberta reporting no current Active Health Advisories related to drinking water.
British Columbia	31	590	Boil water advisories related to public and private systems from five regional health authorities.
Yukon	N/A	N/A	Outside Health Canada mandate, not provided publicly by territory.

Similar data are collected by municipalities. As an example, the evaluators extracted data for the First Nation community of Constance Lake. In determining a comparable community, the evaluators isolated communities within a 500 km radius for similar geography, and narrowed the selection to communities of similar size. Using these parameters the most similar community was Hornepayne Ontario. While the variables collected for the two communities are similar (with slightly different intervals for testing), in both cases, the extracted reports are several hundred pages with specific volume testing on specific dates.

The Canadian Water Quality Index

Many municipalities have, however, employed the use of the Canadian Water Quality Index (CWQI) endorsed by the Canadian Council of Ministers for the Environment.^{Footnote 32} This index was introduced by the Water Quality Guidelines Task Group of the Canadian Council of Ministers for the Environment,^{Footnote 33} and was a modification of the original British Columbia Water Quality Index.

Conceptually, the CWQI comprises three factors. F₁ (scope) assesses the extent of water quality guideline non-compliance over time, and is calculated:

Text alternative for F₁

F₁, representing scope, is equal to the number of failed variables divided by the total number of variables, multiplied by 100.

F₂ (frequency) assesses how often the observed value was off acceptable limits, representing the percentage of individual tests that do not meet the objectives, and is calculated:

Text alternative for F₂

F₂, representing frequency, is equal to the number of failed tests divided by the total number of variables, multiplied by 100.

F₃ (amplitude) assesses the amount by which the failed test values do not meet their objectives, resulting an "excursion_i" variable^{Footnote 34}, which is summed and divided by the number of tests to yield a sum of excursions, or "nse", which is calculated:

Text alternative for NSE

NSE is equal to the sum of excursion, divided by the number of tests.

To yield a value between 0 and 100^{Footnote 35}, the formula is normalised as:

Text alternative for F₃

F₃, representing amplitude, is equal to NSE divided by 0.01NSE plus 0.01.

The CWQI is then calculated as:

Text alternative for CWQI

The CWQI is calculated as follows: CWQI is equal to 100 minus the square root of F₁ squared plus F₂ squared plus F₃ squared, divided by 1.732.

The factor of 1.732 is used to scale the index from 0 to 100. Using this approach, the water quality would then be ranked in one of five categories:

1. Excellent (95-100)
2. Good (80-94)
3. Fair (60-79)
4. Marginal (45-59)
5. Poor (0-44)

The Canadian Council of Ministers for the Environment has released guidelines^{Footnote 36} on site-specific application of the CWQI. While data compiled for the formula for specific sites may be complex, the output of an easy-to-understand index that can be compared between "like" communities has the advantage of providing the means to measure comparability and give First Nations and federal departments a better measure of progress toward this goal. AANDC and Health Canada should be poised to engage water technicians and municipalities in an effort to gather data in a fashion necessary to measure comparability using this index.

System Type

Another measure of comparability would be simply to compare the types of water systems in place for communities of similar size and geography. While these data exist in AANDC database extracts, the comparison may not always be straightforward, as 1) the naming conventions of system types may not be the same (water system expertise would be required); and 2) it may not be appropriate to assume that the comparison community off reserve has the appropriate system in place. Asset and risk variables with AANDC's databases are comprehensive; however it would be difficult to acquire comparable information for similar communities off reserve. The more optimal approach to comparability may be to assess the cost of operations and maintenance off reserve and compare it to the investments in operations and maintenance levels on reserve. Given much of the evidence in this evaluation suggests that much of the risk lies in the maintenance and life cycle of systems, this comparison could inform the Government of Canada with respect to reasonable expectations on the level of investment needed in operations and maintenance to reduce risks and meet life cycle expectancy.

Ultimately, from a point of view of fairness and equity, it is reasonable to expect that individuals living on reserve would have the same sense of trust and safety using their water at tap as other Canadians, and certainly perceptions of water quality are a valid indicator.

Data on wastewater effluents are now collected by Environment Canada via the Government's recent Wastewater System Effluent Regulations^{Footnote 37} The indicators collected as part of the Government of Canada's Effluent Regulatory Reporting Information System should enable the comparison of First Nation and non-First Nation communities with respect to meeting these wastewater regulations. AANDC would need to work with Environment Canada to assess the degree to which this comparison can be made by community.

Should AANDC decide that comparability continues to be a viable and relevant indicator of the success of investments in water infrastructure and capacity building, this concept will need to be better articulated and operationalised with the guidance of water system specialists.

Measurement of Efficiency

Data as they are currently collected from recipient reporting do not sufficiently allow for assessments of efficiency. The current system tracks information on the number of homes in various circumstances, such as type of water and wastewater systems, and there are no detailed financial reporting parameters in Contribution Agreements. In the most recent Performance Measurement Strategy for the Capital Facilities and Maintenance Program (2009), the only water and wastewater indicator related to efficiency was "the percentage of water/wastewater systems meeting general life expectancy." There were notable limitations with respect to the ability to collect data on this indicator, and it was acknowledged that extensive research would need to be done. As an indicator, this item was never fully developed; however, data collected via Asset Condition Reporting System should be able to flush out this indicator as a proxy for life expectancy. Additionally, comparisons of costs of investments in water systems by type and population covered, as well as costs of operations and maintenance, have the potential to benchmark the relative appropriateness of investments so that the Government of Canada can assess value for money.

4.2.3 Efficiency

As discussed above, Health Canada and AANDC currently lack systematic data for measuring efficiencies in the FNWWAP. Additionally, FNWWAP as an expenditure, acts largely as a top-up to existing operating expenditures and Grants and Contributions related to water and wastewater, thus, it is not possible to separate the incremental results of FNWWAP from the broader water and wastewater spending.

It is clear, however, that while the ideal proportion of total expenditures used for operating expenditures (versus Grants and Contributions) have not really been benchmarked for government programming, in AANDC this proportion for FNWWAP was four percent in 2008, seven percent in 2009, eight percent in 2010 and 3.5 percent in 2011 (see Figure 1).

Over the four years, the financial system indicated that Health Canada spent 62 percent of all the resources available, with the percentage spent increasing over time (from 55 percent in 2008-09 to 70 percent in 2011-12) and with full utilization of Grants and Contributions. Key informants noted that not all relevant operating expenditures may be properly coded to this program area, thus total spending in financial reports is likely an underestimation.

Text alternative for Figure 1: FNWWAP Operating Expenditures as a Proportion of FNWWAP Total Expenditures

Figure 1, a line graph, is titled as and depicts the "First Nations Water and Wastewater Action Plan (FNWWAP) Operating Expenditures as a Proportion of FNWWAP Total Expenditures." The x-axis displays the fiscal year, with a range from 2008-09 to 2011-12. The y-axis depicts the dollar value representing program expenditures. Expenditures range from $0 to $160,000,000. Three lines can be seen on the graph to represent: 1) Total FNWWAP Expenditures; 2) FNWWAP Operating expenditures (versus Grants and Contributions); and 3) Operating Expenditures.

The line depicting total FNWWAP expenditures remains more or less constant from 2008-09, hovering between $120 million and $140 million, and dips below $120 million for fiscal year 2010-11. Between 2010-11 and 2011-12, total expenditures peaked from below $120 million to hover between $140 million and $160 million. Essentially, the line representing total FNWWAP expenditures suggests a funding decrease for fiscal year 2010-11, followed by an increase in expenditures for fiscal year 2011-12.

The line depicting FNWWAP operating expenditures, as well as the line depicting FNWWAP operating expenditures as a percentage of total, closely resemble one another inasmuch as they both hover between the zero dollar mark and the $20 million mark. However, where FNWWAP operating expenditures remain more or less constant, "operating expenditures as a percentage of total" increases marginally between 2008-09 and 2010-11, before falling to 2008-09 levels in fiscal year 2011-12.

The operating expenditures for the FNWWAP and water and wastewater in general comprise a small proportion of the total, indicating that the program is operating efficiently from an internal operations point of view.

The increase in operating expenditures in 2009-10 and 2011-12 are largely due to the $9 million spent on the National Assessment. Proportionally speaking, AANDC operational expenditures appear to be reasonably low, suggesting that the internal administration of the FNWWAP accounts for a small proportion of the total, which can be considered a good indicator of efficiency. A similar trend can be observed in examining the total of all AANDC water and wastewater-related expenditures, with operating expenditures representing 3.1 percent, 3.4 percent, 4.2 percent and 2.6 percent of the total expenditures respectively from 2008-09 to 2011-12. With a recent up-tick in total expenditures, operating expenditures did not increase, suggesting that the vast majority of expenses are directed to projects while keeping internal costs relatively low.

While there appear to be some reasonable indicators of efficiency in terms of operating expenditures, the efficiencies of the activities funded within the Grants and Contributions component^{Footnote 38} of the FNWWAP are much less clear. While it is possible to examine the projects and resulting infrastructure by cost (although this would have to be done looking at the total water and wastewater funding, as the incremental contribution of FNWWAP would likely not be obvious given that communities use the summation of their Grants and Contributions to fund various projects), the efficiency and the "cost-effectiveness" would be much less clear without understanding the value of the system in terms of its sustainability, longevity and impacts on health and safety. The infrastructure projects funded through FNWWAP, for example, are relatively new and it is difficult to know at this point their value relative to their cost, as new systems likely are operating at full functionality.

AANDC produced its National First Nations Infrastructure Investment Plan in 2012, which outlined plans for life cycle management of AANDC-funded community infrastructure, a resource allocation process and an investment planning process. The priority areas included the protection of health and safety as well as assets; health and safety improvements; recapitalisation/major maintenance; and anticipated growth. Regional First Nations Infrastructure Investment Plans are informed by a national priority ranking framework, which are intended to provide a consistent and transparent priority ranking of infrastructure investments. 

AANDC also promotes the use of full life-cycle costing analysis when approving capital construction projects. This type of analysis allows for the consideration of all the associated costs (e.g. for design, construction, operations and maintenance, insurance, and major renovations) of the proposed infrastructure throughout its entire life cycle. This type of information is used to make comparisons between proposed options (e.g. building an expensive installation with low operating and maintenance costs versus a less-expensive facility with higher operating and maintenance costs) and to determine the most cost-effective option in the short, medium, and long term. This exercise is also useful in determining the funding levels required from the First Nation in future years for the operations and maintenance of the proposed infrastructure.

Key to ensuring efficiency includes investing in the proper maintenance of systems, and ensuring the most appropriate systems are in place relative to the capacity needs (referring to water or wastewater system capacity) of the communities. There are also indications that innovative technology may reduce both risk and costs where more than one community shares operator hubs using broad-band for the detection of water issues.

While there is limited extractable data on the uptake of new technologies from recipient databases, one project of note was the Alberta First Nation Technical Services Advisory Group Remote Monitoring Project. Technical Services Advisory Group, which is responsible for the delivery of the Circuit Rider Training Program in Alberta, developed the initiative in direct response to a number of the reoccurring issues discussed above with respect to water treatment plants on reserve, including high staff turnover, improper maintenance and repair, and the troubleshooting and response time. The initiative consisted of the installation of remote water monitoring technology into all the water treatment plants across the province, at a total cost of $4.3 million. These systems are all supported by Technical Services Advisory Group Circuit Rider Program and have been integrated into a province-wide network that links all First Nation schools, administration buildings, health centers, and water treatment plants to the "Alberta SuperNet." Technical Services Advisory Group has also created an in-house Network Operations Center and Help Desk to allow First Nations to have network access with dedicated support.

Currently, plant operators are still responsible for the testing of water quality at the plant along with testing at tap by Health Canada Environmental Health Officers, as per the Guidelines for Canadian Drinking Water Quality. Health Canada provides on-site testing facilities for remote (and the majority of) First Nation communities. The added benefit of the Remote Water Monitoring system is that it would immediately help identify a potential problem and automatically contact the designated personnel.

As another example, the Williams Lake Water Improvement Project was developed in British Columbia at a total project cost of approximately $4.9 million. The water system had been inadequate both in terms of the quantity and quality needed to meet the needs of the approximately 300 people living in the community. The project included the tie-in of two recently drilled wells, the construction of a new water treatment plant, and a new reservoir supply main. The new treatment plant will be equipped with a Supervisory Control and Data Acquisition monitoring system, which will allow remote monitoring and troubleshooting of any issue, including water quality. The community has been on a drinking water advisory since 2006 and it is anticipated the new system will significantly improve the conditions of the drinking water and accommodate continued population growth.