Archived - Summative Evaluation of the First Nations Water Management Strategy

In 2002-2003, Treasury Board authorized Indian and Northern Affairs Canada (INAC) and Health Canada (HC) to implement the First Nations Water Management Strategy (FNWMS or the Strategy) starting in 2003-2004. The authorization expires in March 2008. This report has been prepared to fulfill a Treasury Board requirement to complete a summative evaluation of the First Nations Water Management Strategy to inform renewal of the Strategy's authorities in March 2008.

The Strategy aims to address urgent issues related to drinking water and wastewater in First Nations communities. The Strategy is based on a multi-barrier approach, recognizing that safe drinking water results from several factors: source protection, water and wastewater treatment facilities and operations, operator training and certification, standards and guidelines, and community capacity. Budget 2003 committed an additional $600M over five years to the Strategy: $115.9M administered by HC and $484.1M by INAC. This funding represents up to approximately 25-30 percent of total INAC funding devoted to water and wastewater systems in First Nations communities.

Background

For more than a decade, research has indicated that residents of a large number of First Nations communities do not have access to safe drinking water. A 1995 INAC review of available data, for example, suggested that one in four water systems posed significant risks to human health. Despite investments of more than $560M in system upgrades between 1995 and 2001, along with annual funding of $100-$125M for operations and maintenance, the situation deteriorated. A national assessment of 740 water and 462 wastewater systems in First Nations communities conducted in 2001-2002[NOTE 1]found that approximately one-third of all on-reserve drinking-water systems and one-sixth of wastewater systems posed potentially high-risks to water quality and human health and that there was no adequate schedule in place for testing waterborne contaminants in on-reserve water distribution systemsThe FNWMS was developed by HC and INAC to address existing gaps in water management and the ensuing health risks.

In March 2006, INAC and the Assembly of First Nations announced the Plan of Action for Drinking Water in First Nations Communities in March 2006. The Plan included an investment of an additional $60M over two years to address the full range of issues associated with on-reserve drinking water.

Approach

The evaluators strived to measure:

• progress made in providing safe drinking water and treating wastewater effectively on-reserves from 2003-2004 to 2007;
  
  
• cost-effectiveness of investments in water and wastewater systems and of the program for testing quality and safety of the drinking water on-reserve; and
  
  
• the relevance and appropriateness of the Strategy's approach and the identification of potential improvements.

To assess the overall performance, cost-effectiveness and relevance of the FNWMS, evaluators relied on three principal sources of information: INAC and HC performance monitoring systems and administrative information, onsite inspection reports by third party consultants on behalf of INAC regional offices, and the review of published reports and studies regarding water management issues. Sufficient information was available to report on 20 of the 25 performance indicators listed in the Strategy's Results-based Management and Accountability Framework (RMAF).

Findings

The evaluation determined that while the Strategy has accomplished significant progress toward the immediate, intermediate and final outcomes listed in its RMAF, it has not achieved all of these goals.

Key findings include:

• The number of high-risk water-treatments systems declined from 218 in 2003 to 98 [NOTE 2] in July 2007. Upgrades were completed on 133 high-risk water-treatment systems between 2003-2004 and 2006-2007.
  
  
• The number of systems with maintenance-management and emergency-response plans increased significantly between 2005 and 2007. However, less than 50% of the systems have implemented a Maintenance Management Plan, as required from all systems.
  
  
• The number of First Nations communities participating in the Circuit Rider Training Program increased from 427 in 2003-2004 to 500 in 2006-2007.
  
  
• A severe shortage exists of certified water-treatment system operators in First Nations communities. The percentage of First Nations water-treatment system operators who had completed certification increased from 8 percent in 2003 to 37 percent in July 2007.
  
  
• The number of Environmental Health Officers (EHOs) increased across the country from 70 in 2002 to 96 in 2006. Furthermore, five of seven regions had hired adequate numbers of EHOs.
  
  
• The number of First Nations communities with access to a Community-based Water Monitor (CBWM) increased from 272 in 2002 to 592 in 2006. In addition, the percentage of First Nations communities with access to portable kits for onsite bacteriological analysis of drinking water increased from 56% in 2002 to 91% in 2006.
  
  
• The number of water samples taken has increased seven-fold since 2002. However, compliance with the frequencies standards for bacteriological, chemical and radiological parameters outlined in the Guidelines for Canadian Drinking Water Quality (GCDWQ) still requires improvement.
  
  
• The Protocol for Safe Drinking Water in First Nations Communities, which includes a consolidation of all the standards and guidelines applicable to the design, construction, operation, maintenance and monitoring of drinking water systems was published and implemented in March 2006.
  
  
• Rudimentary calculations based on available data demonstrate that investments in operations and maintenance are more cost-effective than investments in capital to reduce the level of risk posed by a water system. Comparison of the FNWMS monitoring approach to an alternative approach illustrated that resources (i.e., CBWMs and testing kits) were secured at a lower cost. However, evidence suggests that HC program expenditures are not adequately linked to program outputs and outcomes to fully measure value for money.
  
  
• INAC exercises a role of de facto regulator responsible to verify the effective implementation of the standards and guidelines defined in the protocol. This role is not compatible with is function as the main funding agency for water and wastewater systems and its accountability regarding the overall results of the Strategy.
  
  
• INAC should promote the use of cost-effective, small-scale systems such as private wells and septic tanks.

Despite the Strategy's accomplishments, the water and wastewater systems on many First Nations communities continue to be inadequate and continue to pose undue health risks. A clear and urgent need to improve the quality of water available in First Nations communities continues to exist. Considering the results and measurable progress accomplished under the Strategy, the evaluation concludes that the overall approach remains relevant, should continue and be improved.

Recommendations

Indian and Northern Affairs Canada

1. The Department should continue to provide assistance to First Nations for the building, operation, and maintenance of their water and wastewater systems as well as for the training of system operators. The Department should commit to address, in the short term, the remaining major risk issues with water systems.
   
   
2. Monitoring and reporting practices should be enhanced to ensure that the First Nations and INAC have reliable information about drinking-water and wastewater systems in First Nations communities. In particular, INAC should:
   
   
   1. ensure that all funded systems undergo complete annual on-site inspections according to the "Guide for Annual Inspections of First Nations Drinking Water Systems" in the Protocol for Safe Drinking Water in First Nations Communities;
      
      
   2. report more clearly on the extent to which systems meet established design, construction and water-quality standards; and
      
      
   3. collect data that support basic cost-effectiveness measurement, e.g., cost per connection and cost per person served for capital expenditures and for operations and maintenance.
      
      
3. INAC should
   
   
   1. revise its funding agreements with First Nations to ensure that funds awarded for operation and maintenance are used for that purpose; and
      
      
   2. take measures to ensure that the Maintenance Management Plans and Emergency Response Plans required under the Protocol are in place
      
      
4. INAC must significantly improve operator training and certification by substantially upgrading the Circuit Rider Training Program or by implementing a superior alternative.
   
   
5. The Department must address gaps in program design with a view to providing support for alternative cost-effective solutions such as wells and septic tanks.
   
   
6. Future policy development should consider the implementation of a regulatory framework that would separate INAC's roles as funding agency and de facto regulator. The option of having a separate federal department/agency or the provinces/territories or other entity (such as a First Nations organization/institution or aggregation), other than INAC, to enforce regulations for water and wastewater on-reserve should be explored.
   
   

Health Canada

Delivery Capacity

1. HC should develop recruitment and retention strategies to ensure that there is an adequate number of EHOs and CBWMs across the country to deliver core environmental health programming in all First Nations communities south of 60 degrees, since there are existing EHO vacancies, anticipated retirements, and the high turn over of EHO and CBWM positions.
   
   
2. Given the limited data collected on the distribution of standards and protocols to EHOs within the regions, HC should work with the REHMs to collect more targeted information related to the distribution of standards and protocols produced by HC to assist with enhancing delivery capacity.
   
   

Frequency of Monitoring

• 3.HC should undertake a review to identify the barriers preventing all communities, in all regions, from completing the required frequency and quality control standards as per the best management practices and the GCDWQ.
  
  
• 4.Quality control measures should be incorporated into a fully articulated quality assurance system. A strong QA system requires strong management support, clearly defined responsibilities and reporting requirements at each level of operations, detailed standard operating protocols, and a rigorous audit system to ensure compliance. Any QA system would require cooperation and support from staff at the national, regional and community levels to set out agreed upon roles and responsibilities.

Identification of Drinking-Water Quality Problems

• 5.HC should re-examine its Performance Measurement Strategy, associated outcomes, performance indicators and data sources. Specifically:
  
  
  1. HC should re-examine its outcomes, performance indicators and data sources. The Performance Measurement Strategy was based on information available at the onset of the FNWMS. Some outcomes were not sufficiently measurable and were not consistent with the theory underpinning the water monitoring program. Consequently, indicators were subsequently added to measure the success of the DWSP and support this evaluation. As well, whenever possible, performance indicators (e.g., sampling frequencies and QC compliance) should be directly gathered through centralized regional databases. A revised Performance Measurement Strategy would include appropriate and measurable outcomes and performance indicators as well as reliable and sufficient data sources to support program development and refinement.
     
     
  2. HC should improve its EHO Questionnaire to monitor progress in ensuring safe drinking water.
     
     
  3. HC should continue its work to identify and address waterborne threats to human health in First Nations communities. This work would allow HC to better identify and report on instances of disease outbreak as well as develop tools and materials for communities to reduce the health risk associated with waterborne threats.
     
     

First Nations Community Confidence in Drinking Water

• 6.HC should explore the issue of First Nations communities' confidence in their water.

Value for Money

• 7.HC should strengthen links between the Performance Measurement Strategy and program expenditures to support future value for money exercises and evaluation.

This report has been prepared to fulfill a Treasury Board requirement to complete a summative evaluation of the FNWMS to inform renewal of the Strategy's authorities in March 2008.

Background

A review by INAC in 1995 of available data on the quality of drinking-water systems showed that one in four water systems posed significant risks to the health of people living on-reserves. As a consequence, INAC invested more than $560 million between 1995 and 2001 to undertake urgent water and wastewater system upgrades. This is in addition to the $100 to $125 million provided annually by INAC to operate water and wastewater facilities in First Nations communities.

In 2001-2002, INAC conducted a series of onsite inspections of all water and wastewater systems on-reserves; these inspections revealed extensive problems with water and wastewater systems in First Nations communities. A gap assessment on the delivery of the Drinking Water Safety Program (DWSP), completed by Health Canada (HC) in 2002, demonstrated that drinking-water quality monitoring in First Nations communities was insufficient to protect human health.

In 2002-2003, INAC and HC received authorization to implement the First Nations Water Management Strategy (FNWMS, or the Strategy), starting in 2003-2004. To support the Strategy, the Budget 2003 committed an additional $600 million over five years; $115.9 million to be administered by HC and $484.1 million by INAC. This funding was expected to represent up approximately 25 to 30 percent of INAC total funding targeted at water and wastewater systems in First Nations communities.

In March 2006, the Minister of Indian Affairs and Northern Development and the Assembly of First Nations announced the Plan of Action for Drinking Water of First Nations communities (the Plan of Action) to address the most urgent issues related to drinking water in First Nations communities.

Roles and Responsibilities

Responsibility for the management of drinking water in First Nations communities is shared between First Nations and the federal government.

First Nations are responsible for planning, designing, constructing, operating, and maintaining their communities' infrastructure facilities and services, such as water and wastewater treatment, and distribution and collection facilities. Once a system is in place, the First Nations community must ensure it is properly operated and maintained, that it is tested on a regular basis and that appropriate training and education are provided to system operators. Additional support is provided by tribal councils, First Nations technical services units, professional engineering consultants and others.

The federal government provides assistance to First Nations to support them in providing safe drinking water to residents. Federal programs and funding related to drinking water on-reserves are based on government policy adopted in the 1960s and 1970s, and on parliamentary appropriations.

INAC provides funding to help First Nations communities supply water services on-reserves. It also contributes to the cost of the operation and maintenance (O&M) of water and sewer facilities on-reserves and for the training and certification of staff, including water-treatment plant operators.

Focusing on preventive activities, HC works with First Nations communities south of 60 to identify potential drinking-water quality problems. HC provides funds to First Nations to sample and test drinking water, and review, interpret and disseminate the results. HC also funds the training of Community-based Drinking Water Monitors (CBWMs), and, in collaboration with the provinces and territories, maintains the Guidelines for Canadian Drinking-Water Quality (GCDWQ).

Territorial governments are responsible for drinking water in Yukon, Northwest Territories and Nunavut.

The First Nations Water Management Strategy

INAC and HC developed the FNWMS to address urgent drinking water and wastewater issues in First Nations communities and the ensuing health risk. The initiative resulted, in part, from the findings of the Walkerton Commission of Inquiry [NOTE 3] and from the findings of the national assessment of water and wastewater systems in First Nations communities conducted in 2001-2002 and published in 2003. [NOTE 4] INAC conducted onsite risk assessments of 740 water and 462 wastewater treatment systems in First Nations communities. The assessments concluded that approximately one-third of all on-reserve drinking-water systems and one-sixth of wastewater systems posed "potential high-risks" to water quality and safety and, therefore, to human health. In addition, HC's assessment determined that no regime was in place to test water-distribution systems.

The FNWMS Approach and Deliverables

The Strategy is based on a multi-barrier approach, as recommended by the Walkerton Commission of Inquiry. The federal framework for a multi-barrier approach is outlined in a brief HC position paper titled From Source to Tap –the Multi-Barrier Approach to Safe Drinking Water ,[NOTE 5] which describes an integrated approach to providing drinking-water protection.

Within this framework, the Strategy outlines a seven-point plan for First Nations water and wastewater to be implemented over five years:

A plan to upgrade and build water and wastewater facilities to meet established design, construction and water quality standards with a priority on identified facilities.

An effective water quality monitoring program combined with a comprehensive and coordinated compliance and reporting regime that will improve the detection of drinking water problems in a timely manner thereby reducing the possibility of risk to health.

An effective and sustainable operation and maintenance (O&M) program designed to ensure the safety of the residents and the protection of the assets with a priority on identified high-risk facilities.

A plan for the continued expansion and enhancement of training programs, to ensure that all operators have the skills, knowledge and experience required to fulfill their responsibilities, supported by the introduction of mandatory certification requirements for all operators.

A set of integrated water quality management protocols with clearly defined roles and responsibilities consistent with national performance standards along with improvements in emergency response procedures.

A public awareness campaign aimed at informing both First Nations decision-makers of their roles and responsibilities in ensuring the safety of water supplies within their communities and First Nations households of measures they can take to protect the quality of water within their home and community.

A comprehensive set of clearly defined standards, protocols and policies, using a multi-barrier approach.[NOTE 6]

These seven elements are in keeping with the recommendations of the Walkerton Commission of Inquiry.

Outcomes and Targets

In compliance with Treasury Board policies on transfer payments and evaluation, a Results-based Management and Accountability Framework (RMAF) was developed for the FNWMS, which provides a road map for this evaluation.

According to the framework, the Strategy is implemented through four key activities and their outputs:

• Construction and upgrades of water and wastewater systems
  
  
• Improvements to operating and maintenance practices
  
  
• Expansion and enhancement of operator training
  
  
• Increasing resources for monitoring and public awareness and development of standards and protocols

Table 1. FNWMS key activities and outputs

According to the framework, these activities and outputs are expected to produce the following immediate outcomes:

• Upgrades to high-risk water and wastewater facilities identified in 2002-2003
  
  
• Enhanced O&M practices
  
  
• An increase in the number of operators trained under the Circuit Rider Training Program
  
  
• An increase in the number of certified operators
  
  
• Improved and consistent delivery of water-monitoring programs
  
  
• Increased frequency of water-quality monitoring on-reserves based on the Guidelines for Canadian Drinking-Water Quality [NOTE 7]

The combined impact of these outputs and immediate activities are to lead to the following intermediate outcomes:

• Water and wastewater facilities meet established federal standard and guidelines
  
  
• Increased ability to identify drinking-water quality problems and potential waterborne diseases

These outcomes should be achieved by year 4 (2006-2007) or 5 (2007-2008) of the Strategy.

In the long term, the Strategy is expected to have the following impact:

• Reduced health risks associated with water quality and supplies issues
  
  
• Increased capacity of First Nations communities to address potential water-quality problems
  
  
• Increased in First Nations communities confidence in their drinking water

According to the framework, the intermediate outcomes would be achieved after the first three years of the Strategy's implementation. More precisely: "in year 3 of implementation of the FNWMS, all high-risk water and wastewater systems will meet established federal standards and guidelines". [NOTE 8] However, this commitment is not consistent with other wording that predicts that intermediate outcomes will be achieved in year 4 and 5. Nevertheless, at the end of the Strategy, all high-risk systems should have been eliminated. At that time, the "established federal standards and guidelines" referred to in the framework were the GCDWQ and the Guidelines for Effluent Quality and Wastewater Treatment at Federal Establishments (1976). However, the Strategy committed to delivering a comprehensive set of clearly defined standards based on the multi-barrier approach.

Delivery model

The $600 million of additional money allocated to the Strategy is intended to supplement existing program resources within INAC and HC.

The FNWMS at INAC

Water and wastewater systems in First Nations communities are funded by INAC's Capital Facilities and Maintenance Program (CFMP). This program transfers funds to First Nations communities who are to provide community services such as water and sewage, schools, roads, fire protection, and community buildings. CFMP's objective is to improve living conditions on-reserves.

INAC, in consultation with First Nations, sets priorities for the allocation of funds and defines the conditions for their use. For drinking-water systems on-reserves – including water-treatment plants, water intake, pipes, and water trucks –CFMP covers the full cost of the design, construction acquisition, upgrades and major repairs to on-reserve water systems that supply residential and community buildings. The costs of building and maintaining individual wells, businesses and water systems servicing fewer than five houses are not covered. INAC approves projects and funding according to departmental guidelines but must obtain Treasury Board approval for project budgets over $15M. INAC also monitors compliance. The actual level of water-distribution services funded by INAC varies according to the housing density and hydro geological conditions of a community. The service level can vary from piped water, to delivery of water by truck, to cisterns. [NOTE 9]

The CFMP also provides funding to First Nations communities for the operations and maintenance (O&M) of eligible water systems. INAC funds 80 percent of the estimated operations and maintenance costs of water systems; First Nations communities are expected to cover the remaining 20 percent through user fees or other revenues. First Nations operators, under the direction of their Chiefs and Band Councils, are responsible for regularly sampling and testing the raw, treated and distribution-system water. INAC also provides funding for training and certification of system operators. In addition, INAC covers 80 percent of the costs when First Nations communities buy drinking water from neighbouring municipalities.

Incremental funding from the Strategy is to be applied to the following:

• Urgent water and wastewater infrastructure projects in First Nations communities, including engineering studies and construction/upgrading of water and wastewater systems, with priority given to high- and medium-risk systems identified in the National Assessment.
  
  
• Improvements in operations and maintenance practices, and in the monitoring of increasingly sophisticated water and wastewater-treatment systems.
  
  
• Assistance with the preparation and implementation of maintenance-management and emergency-response plans.
  
  
• Increased training opportunities for operators and improved access to CRTP to facilitate certification.
  
  
• Enhanced management activities, including the addition of staff at INAC Headquarters and regional offices and at Public Works and Government Services Canada. Staff carry out the following activities:
  
  
  • Implement action plans for recommended improvements to high- and medium-risk water and wastewater systems.
    
    
  • Develop clearly defined quality standards and protocols.
    
    
  • Design and plan reviews and inspections during construction and commissioning.
    
    
  • Establish a consistent and appropriate monitoring regime for drinking water and wastewater and ensure it complies with federal guidelines.
    
    
  • Continue to develop information systems that track progress.
    
    
  • Develop procedures to manage the Strategy.
    
    
  • Develop and implement a public-awareness campaign targeted at First Nations communities.

The DWSP and FNWMS at HC

Through the Drinking Water Safety Program (DWSP), HC works with First Nations communities south of the 60th parallel to ensure drinking water is monitored according to the GCDWQ. More specifically, out of the approximate 392,000 people living on-reserve (INAC, 2006), HC currently works with 682 First Nations communities to monitor their drinking-water quality in Community Water Systems, Public Water Systems and Trucked Water Systems.

These three water distribution systems are different by design and provide water to the majority of people living on-reserve. A Community Water System is a public drinking-water distribution system with five or more connections to houses or other buildings. The estimated population on-reserve receiving water from this type of system is 236,036 (INAC 2006). A Public Water System is a drinking-water distribution system with fewer than five connections that provides drinking water to a federal or First Nations operated public facility, such as a daycare. In addition to the public facilities served, an estimated 15,512 people living on-reserve get their water from this type of water system (INAC, 2006). A Trucked Water System is a public water distribution system with five or more connections (to houses or other buildings), which provides drinking water by truck from a water treatment/storage facility to user destinations; such as cisterns and barrels, equipped with a truck fill. The estimated population served is 59,192 (INAC, 2006).

HC's delivery of the DWSP is carried out through a team approach, which focuses on community capacity building in order for First Nations communities to be able to monitor their own drinking-water quality. Community-based Water Monitors (CBWMs) are the core component of the DWSP. They are community members, funded by HC, to sample and test drinking water for potential bacteriological contamination as a final check on the overall safety of drinking water at tap. If a community does not have a CBWM, an Environmental Health Officer (EHO), who is a certified public-health inspector employed by HC or a First Nations organization, samples and tests the drinking water with permission from the community.

EHOs test drinking-water quality for chemical, physical, and radiological contaminants and maintain quality assurance and quality control. EHOs also review and interpret drinking-water quality tests, disseminate the results to First Nations communities, deliver training to CBWMs, supply educational materials to communities and provide recommendations to First Nations and INAC. In First Nations communities where responsibility for environmental-health programs has been transferred to the community, First Nations stakeholders are responsible for monitoring the quality of their drinking water.

A gap assessment on the delivery of the DWSP was completed in 2002 by Health Canada, which showed that a testing schedule for waterborne contaminants in on-reserve water distribution systems was not in place. For example, monitoring of bacteriological contaminants in distribution systems only met, on average, 29% of the frequency recommended in the GCDWQ. Therefore, HC's assessment indicated that drinking-water quality monitoring in First Nations communities was insufficient to protect human health.

The First Nations Water Management Strategy (FNWMS) was developed by HC and INAC to address existing gaps in water management and the ensuing health risks. For HC, the additional funding was intended to improve the implementation of the DWSP. Specifically, by the end of year five of the Strategy, HC was to monitor all distribution systems with five or more connections and cisterns as per the GCDWQ and according to best management practices.

The incremental funding from the Strategy is to accomplish the following:

• Increase capacity in First Nations communities to monitor drinking-water quality at tap in all distribution systems and cisterns south of the 60th parallel. Funds are used to hire and train CBWMs in areas where it is difficult or impossible to get samples to an accredited laboratory in a timely manner.
  
  
• Purchase, for all communities, kits to test for E. coli, coli form, Giardia and Cryptosporidium. The kits facilitate onsite testing and the early detection of drinking-water quality problems.
  
  
• Purchase lab kits and pay for tests conducted in accredited laboratories.
  
  
• Increase the number of EHOs who monitor drinking-water quality according to GCDWQ. EHOs review, interpret and disseminate the results of drinking water tests, train CBWMs and develop local education and awareness programs.
  
  
• Maintain an early-warning database, analyze incoming data, identify potential problems, publicize results, and respond to inquiries.
  
  
• Support research into waterborne diseases and potential contaminants.

Environment Canada

Environment Canada (EC) also plays a role in First Nations water and wastewater systems. EC develops guidelines, standards and regulations on the protection of source water. The Guidelines for Effluent Quality and Wastewater Treatment at Federal Establishments (1976), for example, establishes parameters for treated wastewater released into streams, rivers and lakes. EC is currently developing a long-term strategy to manage risks associated with wastewater effluents on federal property, including reserves.

Funding

INAC allocates approximately $300M annually from the $1.2 billion of the Capital Facilities and Maintenance Program for water and wastewater infrastructure. Approximately $220M of this amount is spent on capital investments, while the remaining $80M pays for operation and maintenance (including $8M for operator training, education and certification). The Strategy provides approximately $100M of the $300M annually from 2003-2004 to 2007-2008. The remaining $200M comes from the Department A-base.

Apart from the funding received for the FNWMS, HC invests approximately $5M annually in monitoring programs in First Nations communities south of 60 degrees. In 2002, HC invested an additional $5M to protect and enhance drinking-water quality on-reserves. This included $4.4M in Treasury Board funding and $600,000 from internal re-allocations.

From 2003-2004 to 2006-2007 INAC spent a total of $296,787,382.72 from the Strategy funds, and expects to spend an additional $105,681,800.00 in 2007-2008 for a total of $402,469,182.72. During the same period, a total of $1,030,333,541.01 has been spent on water systems by INAC and it is estimated that in 2007-2008, an additional $675,491,300.00 will be spent on First Nations water issues for a total sum of $1,705,824,841.01. Therefore, from 2003-2004 to 2007-2008, the Strategy represented 24 percent of the total expenditures, actual and planned.

Between 2002-2003 and 2006-2007, HC spent a total amount of $39.9M on the Strategy and expects to spend an additional $19.8M during 2007-2008. Actual spending is under estimated because resources from some regions and headquarters were not all properly coded.

Table 2: Total INAC Resources allocated to First Nations Water Management (millions of $)

Table 3: Total HC Resources allocated to First Nations Water Management (millions of $)

Table 4: Expenditures on water systems (2003/04 to 2006/07) and estimates for 2007/08 by region

Table 5: Expenditures (2002-2003 / 2006-2007) and estimates (2007-2008) from the First Nations Water Management Strategy (FNWMS)

Table 6: HC's expenditures ($K) for the FNWMS

Evaluation approach

Planning for the evaluation began in January 2007 [NOTE 10].

To prepare for the evaluation, key issues were identified, along with relevant background materials, policy documents and programs databases.

Research suggested that information reportedly available from INAC and HC monitoring systems would be adequate to compare results against performance indicators listed in the Strategy's RMAF. [NOTE 11] Although some information was missing, there would be sufficient information to measure achievement of the Strategy's immediate and intermediate outcomes, along with some of its final outcomes. At this stage, it was planned that additional research, including a systematic comparison of First Nations communities with similar-sized communities, and interviews with experts and stakeholders, would be required to assess the overall value and relevance of the Strategy.

Conclusions from the preliminary assessment were validated and confirmed with program officials from both INAC and HC.

Evaluation issues

Evaluation terms of reference were approved in June 2007 by INAC's Audit and Evaluation Committee. The Committee requested that the evaluation be completed in time to inform the program-renewal process. As a result, given the tight timelines, it was decided that the evaluation would rely on information collected by the Programs through their monitoring systems, on recently published reports and literature, and on public consultations.

The terms of reference identified the following general assessment topics:

• progress made in providing safe drinking water and treating wastewater effectively on-reserves from 2003-2004 to 2006-2007
  
  
• value for money from INAC's and HC's investments in water and wastewater systems and water monitoring in First Nations communities
  
  
• relevance and appropriateness of the approach developed and implemented through the Strategy, along with potential improvements

The research for the evaluation began in June 2007 and was completed in September 2007.

Data sources

Evaluators referred to the following information sources:

Document review

To complete and interpret information provided by both departments programs performance monitoring systems, evaluators reviewed other documentation, including research reports commissioned by the departments, audits and evaluations, reports to Parliament, program frameworks, working documents and HC's 2002 gap assessment. A complete list is provided in Appendix B.

Information on expenditures

Evaluators also reviewed information on expenditures related to the Strategy. For INAC, financial information came from two sources: reports generated by financial systems, and data sets on expenditures for each community's action plan, generated by staff of the CFMP. HC's financial information came from reports generated by its financial systems and data sets on expenditures from the regions and Headquarters.

INAC Performance-Monitoring Data

Performance information from INAC came from two main sources: the Department‘s "WATERS" database and from annual inspection reports.

WATERS Database

WATERS is the departmental online database used to monitor and report on the risk levels of on-reserve water and wastewater systems. It includes detailed information on water sources, system designs and operations, operators and certification levels, frequency of testing and reporting, and action plans for upgrades. For each system, the WATERS database lists completed and pending upgrades.

The database is populated by INAC's Headquarters Capital Facilities and Maintenance Program managers, based on information provided by INAC regional staff. In 2005, a decision was made to limit information collection to 14 performance indicators. Information on these indicators is available for 2005, 2006 and 2007.

The WATERS database provided only limited information for the evaluation: it does not include the financial information required to assess cost-effectiveness, such as total systems costs, operating and maintenance expenditures, etc. WATERS is not user friendly; it is difficult and time-consuming to generate basic reports in table format [NOTE 12], for example. There are also reliability issues, such as gaps in demographic information and on the number of homes connected to systems.

Annual Inspection Reports

Annual inspection reports were another source available to evaluators.

The Protocol for Safe Drinking Water in First Nations communities, implemented in March 2006, requires that each system undergo a comprehensive annual onsite inspection. Although reports were required for each system in 2006-2007, not all regional offices conducted onsite inspections. Instead, some regions gathered only the information needed to assess the 14 performance indicators referred to above, while others monitored the performance without visiting the sites. In fact, complete inspection reports were available only from INAC's Manitoba and Québec regional offices. This gap in information significantly limits the scope of this evaluation.

HC Performance-Monitoring Data

Environmental Health Officers Questionnaire

Since 2004-2005, EHOs have compiled information on the water supplies and monitoring activities for the communities they serve. The EHOs provide the data annually through an online questionnaire (EHO Questionnaire). The information relates to the performance indicators identified in the Strategy's RMAF and has been a primary source of data for this evaluation.

The EHOs answer questions for each Community Water System (CWS) and Trucked Water System (TWS) and for every Public Water System (PWS). The evaluators removed inappropriate data–such as information on CWS or TWS with fewer than five connections, as these are not part of the FNWMS.

A comparison of data from fiscal year 2004-2005 and 2005-2006 indicated some variance in the set of reported community sites and water distribution systems. At least some of this variation was the result of improved reporting as familiarity with the reporting system increased. It is also possible that some water distribution systems were either physically changed (e.g., a community may have received a new or modified system or shut down an old one) or were reclassified (e.g., the number of connections was revised below or above five). A database is being created to compare data from the same communities and water systems between years, but is not available at this time. The total number of communities and water systems, for each fiscal year are presented in Table 7 below.

It should be noted that this evaluation report contains several tables with 2002 data. These data were derived from sources other than the EHO Questionnaire, primarily by the Environmental Health Division through surveys and interviews with Regional Environmental Health Managers and EHOs. The 2002 data were the only available data for the creation of an approximate baseline. Thus, the 2002 data were included in this report to provide some illustration of the state of the water file prior to the commencement of the FNWMS and to allow for a comparison of the commencement of the FNWMS and subsequent years.

Temporal trends for indicators drawn from these data, however, should be interpreted cautiously. The data collection methods employed in 2002 were different from 2004 onwards. It is not known if, or how, the different collection methods may affect reported indicator values. Apparent increases from 2002 to later years may, in part, reflect differences in data collection, as well as actual change in the parameter being measured. Differences among years for 2004 onwards may also be affected by variation in the set of systems for which data were collected and in data completeness. Small changes between years, therefore, may reflect changes in data reporting, as well as actual changes in the parameter being measured. As it is not possible to distinguish between the two sources of change, readers should instead focus on general trend in the indicator across all years.

It should also be noted that in response to recommendations in the Commissioner of the Environmental and Sustainable Development's 2005 report, HC updated the Water Management System for collecting performance indicators data and data standards in collaboration with regional offices. All regional offices reported on performance indicators using this system and HC reported to Parliament in this regard for 2005/06. The review and refinement of data collected will be part of the yearly data gathering process. Provincial and territorial public reporting practices have been reviewed and a national standard for public reporting will be finalized.

Table 7: Summary of Community Sites and Water Distribution Systems Reported on in the EHO Questionnaires by Fiscal Year

Note: At the time this report was written, the data for fiscal year 2006-2007 were still being entered by the EHOs. The submission date for these data is later than previous years because the EHO Questionnaire had questions added to it, which resulted in the need for re-programming and testing of the questionnaire. Nevertheless, it is expected that these data will be collected by the Fall of 2007.

Regional Environmental Health Manager Survey

Starting in 2004-2005, Regional Environmental Health Managers (REHMs) have been asked to complete an annual online survey. The survey records data on staffing levels, outputs listed on regional work plans, the number of contribution agreements with First Nations communities and incidence of waterborne illness.

Ad hoc discussions

The evaluators also obtained additional performance information through informal discussions with staff at HC Headquarters. This information related to the some of the outputs and outcomes outlined in the RMAF.

Scope and limitations of the evaluation

This evaluation report presents a credible assessment of the Strategy's performance against the indicators defined in the RMAF. Although the significant gaps in data precluded detailed analyses, the evaluation covers the key aspects of performance. Information was available to assess 20 of the 25 performance indicators.

A more in-depth assessment of the Strategy's impact will be possible once a detailed engineering review of all systems is conducted.

The evaluation's main limitation is its dependence on existing information. Furthermore, the evaluation did not systematically compare First Nations community water systems with those of comparable non-First Nations communities. Such comparisons would provide relevant information about the cost-effectiveness of the investments made in First Nations water systems.

This section presents findings for each evaluation issue; the subsequent section includes interpretations and conclusions.

The Strategy's Performance

To assess progress since 2003/04, information about the Strategy's outputs and its immediate, intermediate and final outcomes was reviewed and analysed using the performance indicators in the RMAF. It is important to note that the RMAF does not include indicators to assess the Strategy's relevance and cost-effectiveness.

Outputs

Outputs are reviewed following the Strategy's seven points and list of deliverables.

Plan to Upgrade and Build Systems to Meet Standards and Guidelines

The first component of the Strategy is "a plan to build and upgrade systems...with a priority on identified facilities." It is assumed that the "identified facilities" are the high-risk systems identified in INAC's 2003 national assessment of First Nations water systems. The performance indicator for this component is the production of a five-year capital plan and regional action plans. [NOTE 13] The five-year Long-Term Capital Plan (LTCP) for INAC, including major water projects, was produced in 2002-2003 and subsequently updated. Water systems were identified as a priority within the Plan.

However, there are some limitations inherent this type of planning. The main one is that water issues are framed solely in terms of capital management, not in reference to water-quality problems in specific communities. This approach restricts the definition of water management in First Nations communities to the range of activities and equipment covered by the Capital Facilities Maintenance Program. This limitation led members of the Expert Panel for the Standing Senate Committee to state that communities clearly at higher risk failed to appear as high-risk on the Department's risk assessment because they did not have any water systems at all.

Drinking Water Monitoring Program

The second deliverable of the Strategy is "an effective water quality monitoring program combined with a comprehensive and coordinated compliance and reporting regime."

Outputs for this component are the number of EHOs and CBWMs. In 2002, HC calculated the minimum number of EHOs and CBWMs needed to serve First Nations communities adequately and to meet the monitoring-frequency standards included in GCDWQ. Personnel were hired and/or trained accordingly during the first four years of the Strategy.

Number of EHOs Hired

EHOs are the primary implementers of HC's water-monitoring program for First Nations communities. EHOs train CBWMs, maintain quality assurance and quality control, and test drinking water for chemical, physical, and radiological contaminants. EHOs review and interpret tests and disseminate results to First Nations communities. If water quality is unsatisfactory, the EHO immediately communicates the appropriate action (e.g., a boil-water advisory) to Chief and Council for action. As well, EHOs play an important role in quality assurance and quality control.

In 2002, HC and First Nations stakeholders employed a total of 70 EHOs; each held valid Certificates in Public Health Inspection (Canada) issued by the Canadian Institute of Public Health Inspectors. The EHOs delivered Environmental Health Services to more than 600 First Nations communities. Along with water-quality monitoring, these services included house inspections for mould and vector-borne disease surveillance. In 2002, 10 EHOs were dedicated to the DWSP across Canada; their duties included sampling, testing, interpreting, communicating results, and providing advice. HC's 2002 gap analysis indicated that an additional 28 EHOs would be needed to monitor water quality in First Nations distribution systems and cisterns according to GCDWQ standards.

As of March 2006, five of seven regions had hired adequate numbers of EHOs and met the target set for 2008 (see Table 8). Only the Ontario and Pacific regions–home to most First Nations communities–still have vacant positions. The two regions expect to fill the remaining vacancies by the end of 2008.

Table 8: The Number of EHOs in Each Region and Nationally, by Fiscal Year, and the Target Value to be Reached by 2008

Note: The gap analysis completed by HC in 2002 identified the target to meet by 2008. This target is based on the work of the EHO including time to travel to remote and non-remote communities, to monitor drinking-water quality, to train the CBWMs and to do the QA/QC program.

Hired CBWMs

Monitoring water according to GCDWQ frequency standards is particularly difficult in remote and isolated communities. Given the inadequate number of EHOs in place in 2002, many communities had to rely on CBWMs: residents trained to sample and test drinking water for bacteriological contamination.

HC provides funds to Chief and Council to build capacity within the community to monitor drinking water quality through the Community-Based Water Monitoring Program. HC trains CBWMs to sample and test drinking water for potential bacteriological contamination, as a final check on the overall safety of the drinking water at tap. In communities without CBWMs, water quality is monitored by either an EHO or a Certified Public Health Inspector employed by HC or a First Nations.

HC's 2002 gap analysis determined that an additional 290 CBWMs were needed to meet the GCDWQ standard of one monitor per community site.

Counting the absolute numbers of CBWMs hired each year proved more difficult than anticipated for a variety of reasons. The duties of a CBWM rarely constitute a full-time job. In many communities, multiple residents are trained to monitor water quality, and the role of the CBWM is shared among many people. In some cases, water quality is monitored by someone performing another role, such as a Community Health Representative. Some CBWMs service multiple communities. As a result, the numbers of CBWMs hired and trained each year may provide little indication of the number of communities with access to local monitoring services.

The number of community sites with access to CBWMs was calculated based on the EHO Questionnaire. The number of community sites with access to a CBWM has increased steadily since 2002, with 592 community sites having access to a monitor in 2005-2006 (see Table 9).

Table 9: Number of Community Sites with Access to a CBWM by year

Operations and Maintenance

The Strategy's third deliverable was to provide an effective and sustainable operation and maintenance program that would ensure the safety of First Nations residents and protect water and wastewater systems. Funding priority was to be placed on high-risk facilities. The Strategy's RMAF output indicator for this deliverable is the number of approved maintenance and management plans (MMPs). The number of approved plans has not been monitored in the WATERS database. Program officials stated that if a plan is being implemented, it has been approved. [NOTE 14] Information on maintenance and management plans implemented, however, is available and discussed in this report's Immediate Outcomes section.

Training and Certification

The Strategy's fourth deliverable was to expand and enhance operator-training programs to ensure that all operators have the skills, knowledge and experience required to fulfill their responsibilities. This was to be supported, at some point in time, by the introduction of mandatory certification requirements for all operators.

The output indicator for this deliverable in the Strategy's RMAF is the number of facilities with trained operators. This information is not readily available. [NOTE 15 ]However, information on the overall number of trained and certified operators is found in the next section.

Water Quality Management Protocols

The Strategy also committed to deliver a "set of integrated water quality management protocols". The output indicator for this activity is the number of approved and implemented procedure manuals, standards and protocols for the management of water systems and issues. It was the responsibility of HC to produce these documents and report on them.

Development of standards and protocols at the national level

HC developed standards and protocols to improve drinking-water quality monitoring. Many of these standards and protocols address recommendations made in the Commissioner of Environment and Sustainable Development's (CESD) 2005 report regarding the design and construction requirements of water and wastewater treatment plants, and program delivery. Discussions with key program personnel at HC revealed that HC has developed:

• The National Framework for the Review Process of Water and Wastewater Systems in First Nations Communities (National Framework), in collaboration with EC and INAC. This document delineates the roles of each department in the integrated review process for drinking water and wastewater project proposals in First Nations communities. To help HC's First Nations and Inuit Health Branch (FNIHB) regional offices effectively review projects from a public-health perspective, FNIHB-HC Guidelines for the Review of Water and Wastewater Project Proposals in First Nations Communities South of 60 was also created.
  
  
• The Procedure Manual for Safe Drinking-Water Quality in First Nations Communities South of 60 (the Procedure Manual), was published in 2004. Designed for EHOs, the Procedure Manual describes appropriate management practices for monitoring water quality in distribution systems with five or more connections and in cisterns, as per the GCDWQ and From Source to Tap: The Multi-Barrier Approach to Safe Drinking Water. The Procedure Manual was updated in 2006-2007 to reflect changes in policies and in the GCDWQ and re-released as administrative guidelines in 2007.
  
  
• The National Framework for the Training and Evaluation of Community-Based Drinking Water Monitors establishes a standard for CBWMs. The framework builds on best management practices in the regions and is based on the Procedure Manual. This framework has been used for the training Community-Based Drinking Water Monitors.
  
  
• The Procedure for Addressing Drinking Water Advisories in First Nations Communities South of 60° was developed in collaboration with INAC. The document outlines roles and responsibilities, and describes appropriate actions to be taken following DWAs.
  
  
• As part of a larger interdepartmental effort led by HC's Healthy Environments and Consumer Safety Branch, draft procedures to address waterborne illnesses on-reserve were developed. HC identified current procedures used to address potential waterborne and foodborne illnesses in First Nations communities. This remains a work in progress.
  
  
• HC drafted a policy to address the quality of drinking water in individual wells on-reserves and in wells with two to four connections, which are not currently covered by HC's DWSP. HC has begun to develop the materials needed to phase-in this policy.

HC has developed additional tools to support field staff. According to discussions with HC personnel, FNWMS allocated resources to HC to address high and medium-risk communities from a public-health perspective. As a result, HC developed a risk-assessment tool in 2005. The tool enables proactive identification of drinking-water quality problems and helps HC and First Nations assess public-health risks.

The tool considers the risk factors associated with:

• Source Water
  
  
• Treatment Plant Operation
  
  
• Drinking Water Distribution
  
  
• Drinking Water Quality
  
  
• Disease Surveillance

The risk-assessment tool will be used on all piped distribution systems with five or more connections, and on all cisterns in First Nations communities south of 60. The tool is also being used to improve water management in First Nations communities and inform the allocation of funds to high-risk water systems.

In response to recommendations made in the Commissioner of the Environment and Sustainable Development's 2005 report, HC has collected results on initial assessments of community distribution systems and program performance indicator information using the Relative Public Health Risk Assessment Tool. The analysis of these results and program performance indicators will be used to assist the Government of Canada and First Nations communities in decision-making

HC has also begun to address two further issues: cyanotoxins and disinfection by-products. Cyanotoxins are associated with blooms of blue-green algae and can be toxic to humans and animals. Given that the tests for measuring cyanotoxin levels can be expensive and time-consuming, HC believes communities should have access to an alternative. To this end, HC has developed appropriate laboratory methods and has begun to evaluate kits currently on the market. One of the kits is now ready for pilot testing in First Nations communities.

The second issue involves disinfection by-products (DBPs) in treated drinking water. Chemicals such as chlorine are added to source water to eliminate hazardous pathogens. Some of these chemicals, however, interact with organic materials in source water and form by-products that can be harmful to health. Of the several DBPs identified to date, two types are of particular interest: nitrosamines and the mutagen X (MX) analogues. HC has identified and optimized analytical methods for these and 27 other DBPs. Methods are now ready to conduct analysis of water samples from First Nations communities to assess exposure and estimate human health risk.

Number of approved and implemented procedure manuals, standards and documents at the regional level

Staff in HC Headquarters develop national policies and guidelines, while regional staff implement DWSP and communicate new procedures and standards to EHOs and CBWMs. Procedure manuals are defined as documents that describe actions necessary to protect drinking-water quality (e.g., the Procedure Manual). Standards are targets and parameters used to measure water quality (e.g., GCDWQ's maximum allowable concentrations for human health or aesthetic objectives). Procedure manuals and standards come from a number of sources including HC, INAC, and the World Health Organization.

Table 10 lists the numbers of procedure manuals, standards and protocols distributed by REHMs to EHOs by region, which has increased in most regions since 2002. The survey question used to gather the data was relatively ambiguous, which may explain the regional variations. Multi-year comparisons within each region appear to be valid, however, as the same REHMs were employed throughout the period. Within most regions, there was little difference between the figures for 2004-2005 and 2005-2006. The large overall increases indicate that REHMs distributed progressively more information to the EHOs.

Table 10: Number of Procedure-manuals, Standards or Protocols Distributed by REHMs to EHOs

Note: * In 2004-2005 Ontario reported a high number of documents (N=650). This outlier was removed from the 2004-2005 data and the 2005-2006 data is believed to be a more accurate result.

Public Awareness Campaign

This section reports on public awareness activities undertaken both at INAC and HC.

INAC has produced two main outputs to deliver on its commitment to raise public awareness:

• A Chief and Council Information Kit composed of four documents produced and distributed in 2004-2005. These documents [NOTE 16] describe the roles and responsibilities of Chief and Council in the O&M of water systems. No formal monitoring was conducted to determine the reach and impact of this campaign. However, a second printing of the document was necessary to accommodate demand, which suggests that the material was positively received and perceived to be useful.
  
  
• A school information kit, Water is a Treasure was[NOTE 17] produced in 2007 for children in Kindergarten to grade 6. The key themes are: the value of protecting water now and for future generations; the wise use of water in daily activities; how water-treatment facilities in First Nations communities ensure safe drinking water; the importance of water operators and the role they play; and the importance of clean, safe and reliable water from its source to the tap and back to the source. The kit includes a poster, a variety of educational activities and a list of Internet resources. A total of 5,147 kits were distributed to band-operated schools (two per band) and 1,040 kits (two per school) were sent to federal- and provincial-operated schools attended by First Nations children. Copies were also sent to Chief and Councils.

HC has begun to develop public-awareness materials to help Chiefs, Councils and EHOs communicate drinking-water advisories. The development of these materials was informed by public opinion research conducted in 2005-2006 with EHOs and First Nations residents on the effectiveness of DWAs.

This research suggested that DWAs were not well communicated, were not systematically followed, and did not have the intended effect of persuading First Nations people to avoid drinking unsafe water. The research also informed plans to improve the communication of DWAs.

Key research findings include:

• Slightly more than half of all respondents (51.0 percent) indicated that they do not use the community water supply, because they either do not trust it, feel it was unsafe (51.9 percent) or did not like the taste of the community water.
  
  
• Less than half of all respondents (42.9 percent) indicated they knew why the advisory was issued and 60 percent said they would follow the advisory if they knew why it was issued. In contrast, only 40 percent followed the advisory if they did not know why it was issued.
  
  
• The most popular method of notification was door-to-door written notification (87.2 percent). The second-most popular method of notification was radio announcements.
  
  
• Respondents were not certain when they needed to use boiled water. For example, 76 percent of respondents still used un-boiled tap water when brushing their teeth.
  
  
• Door-to-door canvassing would be the most effective way to communicate advisories to First Nations residents.
  
  
• Fact sheets and door posters would be the recommended communication tools to provide to Chiefs and Councils to help communicate DWAs.

The research demonstrated that risk-communication materials were needed to change knowledge, attitudes and behaviours, and better protect public health.

HC has developed a suite of materials with an anticipated distribution of winter 2008. The materials include:

• Door hangers (Z-Card pamphlets) featuring text and graphics to convey DWA messages.
  
  
• Public-service announcements.
  
  
• A toolkit for Chiefs and Councils with the materials described above, along with poster and easy-to-understand instructions on how to effectively communicate DWAs to residents using the materials.

INAC Standards, Protocols and Policies

The Strategy's final deliverable is a comprehensive set of clearly defined standards, protocols and policies based on the multi-barrier approach outlined in the Government of Canada's 2003 framework From Source to Tap –the Multi-Barrier Approach to Safe Drinking Water.

This commitment was addressed with the introduction in March 2006 of the "Protocol for Safe Drinking Water in First Nations communities: Standards for Design, Construction, Operation, Maintenance, and Monitoring of Drinking Water Systems". As its title indicates, the protocol provides standards and guidelines for the key aspects of water management on-reserve. The protocol's next iteration will include HC and INAC's "Procedures for Addressing Drinking Water Advisories", and EC's guidance to protect source water.

Immediate outcomes

This section presents the progress to date on each of the Strategy's immediate outcomes. It presents information gathered systematically by the Capital Facilities and Maintenance Program through its ongoing monitoring and information gleaned from program documentation and external documents.

Upgrades Made to High-risk Systems

The first immediate outcome identified is "upgrades made to high-risk water and wastewater facilities on-reserve, as identified in 2002". The performance indicator for this outcome is the number of upgrades made to high-risk water and wastewater facilities. However, the RMAF failed to define what constitutes an upgrade. To measure performance, evaluators used a proxy: the total number of systems that received at least one capital expenditure during the evaluation period. [NOTE 18]

Between 2003/04 and 2006/07, there were 753 water systems that received a risk rating. Over that period of time, 235 of those systems were assessed as being at high-risk, of which 133 received an upgrade (defined as a capital expenditure).

Of the 351 medium-risk systems, 124 received upgrades as did 53 of the low-risk systems.

The average capital expenditure was approximately $1.3M on high-risk systems, $1.6M on medium-risk systems, and $1.8M on low-risk systems.

Over the period 2003/04 to 2006/07, there were 484 wastewater systems in operation. Of the 87 systems rated as high-risk during that period, 31 received upgrades; 62 of the 232 medium-risk received upgrades, as did 40, or 21 percent of the 187 low-risk systems.

Overall, the high-risk systems received the major part of the capital expenditures for wastewater systems.

These numbers show that the proportion (56 percent) of high-risk water systems that received capital upgrades was higher than the proportion of medium (35 percent) or low (32 percent) risk systems receiving upgrades. It also shows that, overall, more funding went to medium-risk-systems than those identified as high-risk.

Table 11: Capital expenditures per water system with an action plan

Table 12: Capital expenditures per wastewater system with an action plan

Operating and maintenance practices

The Strategy's second immediate outcome was "enhanced operating and maintenance practices". Performance indicators for this outcome were originally defined as "the number of maintenance management plans implemented" and "the number of wastewater and water facilities that are at low-risk." The latter assumes that a system with inadequate operations and maintenance would automatically be rated either as a medium- or high-risk system.

When the set of 14 key performance indicators was introduced in 2005, the Strategy's performance indicators were redefined as "the number of maintenance management and emergency response plans in place". Data provided for the last three years shows that the number of systems with maintenance-management plans increased from 71 (22 percent) in 2005, to 128 (35 percent) in 2007. The number of systems with emergency-response plans increased from 129 (15 percent) in 2005 to 154 (17 percent) in 2007.

Table 13: Number of Water & Wastewater Systems with Maintenance Management Plan (MMP) 2005-2007

Table 14 : Number of Systems with Emergency Response Plans (ERP) 2005-2007

Another indication of the state of operations and maintenance practices in First Nations water systems is provided in a set of 148 remedial-action plans listed in Appendix B of the Progress Report on the Plan of Action for Drinking Water in First Nations Communities (Progress Report) tabled in the House of Commons in December 2006. More than half (76) of the 148 high-risk water systems listed are reported to have inadequate operations and maintenance practices.

A number of studies and reviews in recent years have commented on the operation and maintenance of First Nations community water systems.

The 2006 program-led Review of the Capital Facilities Maintenance Program concluded that the federal government's investment in First Nations infrastructure is at risk due to inadequate maintenance of infrastructure.

In a 2005 report, the Commissioner of the Environment and Sustainable Development observed that, "INAC does not know if all funds for operation and maintenance are used for this purpose. The Department provides First Nations about $45 million annually to support the operation and maintenance of their water systems (including wastewater). Under the applicable funding conditions, First Nations have the flexibility to use O&M funds for other purposes, and INAC has limited assurance that they are used for the purpose intended. For the duration of the FNWMS, INAC has raised O&M funding by over 50 percent, to about $75 million. The additional funds are transferred to First Nations under different conditions and cannot be used for purposes other that the operation and maintenance of water systems."

A 2006 program-led review supported these conclusions. It said that, "a review of the audited financial statements of a selection of First Nations and a review of a sample of infrastructure and maintenance reported in INAC's assets conditions database, as well as anecdotal information by regional and First Nations technical advisors, suggests First Nations assets are not fully benefiting from maintenance pursuant to the funding formula INAC employs in minor capital allocations. While a portion is likely due to the inability of First Nations to collect user fees and apply these to their share of the asset operation and maintenance gross funding requirement, the evidence also implies that the assets may not be receiving the full allotment of INAC's share, the net funding allotment." A review of annual inspection reports available to the evaluators confirms that in some cases Bands have not budgeted for operations and maintenance and in a number of cases where there are budgets for operations and maintenance; inspectors have determined that the budget is insufficient.

At hearings held by the Expert Panel in 2006, questions were raised about the Capital Facilities and Maintenance Program's funding formula for operations and maintenance of First Nations water systems. First Nations representatives emphasized that, for many communities, finding funds to cover the portion of operations and maintenance not provided by INAC, i.e. 20 percent, is a "hardship". INAC officials acknowledged to the Panel that the funding formula may need updating and that it is not applied consistently across the country.

Increased number of certified operators

The Strategy's performance indicators for this outcome are an increase in the number of First Nations participating in the Circuit Rider Training Program and an increase in the number of certified operators.

First Nations Participating in the Circuit Rider Training Program

This Circuit Rider Training Program (CRTP) was introduced in the mid-1990's to improve operation and maintenance of water facilities. Under the CRTP, qualified instructors travel to First Nations and train operators onsite. The Program is funded by INAC and is delivered from regional offices.

In 2003, 427 First Nations communities, not including those in Québec participated in the CRTP. In 2007, that number increased to 500. In 2007, all First Nations communities in British Columbia, Saskatchewan, Manitoba and Québec, participated in the training program. Ontario, where approximately First Nations 100 communities do not yet have access to the training program, has the lowest rate of participation.

The 2006 Plan of Action expanded CRTP to make it available to all First Nations communities across Canada.

In a 2005 report, the Commissioner of the Environment and Sustainable Development (CESD) said that, in many instances, much of a CRTP trainer's time is spent troubleshooting rather than providing training to system operators. This claim was substantiated in some of the annual-inspection reports reviewed by evaluators.

In March 2007, a national workshop for CRTP trainers was held in Ottawa. Participants were asked to identify the effective and ineffective aspects of program. Unlike the CESD, trainers viewed troubleshooting as a positive aspect of their work. The workshop's comprehensive notes do not mention that troubleshooting limits the program's effectiveness. Participants' main criticism was that there are not enough trainers to meet demand and liability-insurance requirements. It was also reported that, in some regions, program effectiveness was diminished because trainers were not covered by liability insurance. [NOTE 19]

Workshop participants were asked to estimate the ideal numbers of trainers and annual visits per community. There are 44 trainers; the amount needed to provide the ideal number of annual visits was estimated at 61. The Expert Panel Report mentions that although the training program is funded by INAC, "in most cases the level of support was not adequate to provide as much help as operators would like."

Based on information provided by region staff, the annual cost of the training program is approximately $4.7 million. In some regions not all communities are covered; in other regions there is a higher than optimal ratio of trainers per community. It was suggested that full coverage will require an additional $2.1 million for a total of $6.8 million.

Table 15: First Nations Participating in Circuit Rider Program per Region

Table 16: Actual Circuit Rider Trainers per First Nations community per Region

Table 17: Estimated Ideal Circuit Rider Trainers per First Nations community per Region

Certification of Operators

Progress reports show that the CRTP has brought about significant increases in the number of water-treatment system operators who have achieved the first level of certification or better. The number of certified operators rose from 8% of all operators in 2003 to 37% in July 2007.

Results from the WATERS database differ somewhat. It shows that in 2006, 36 percent of operators were certified. [NOTE 20] By July 2007, the number had risen to 43.4 percent. These numbers do not indicate whether or not the operators have the proper level of certification to operate their system; they indicate only whether operators have level I certification.

These findings show that while there has been some progress, the Strategy overall has not achieved its target of ensuring that all systems are operated by certified operators. It must be mentioned, however, that results differ by region; in Saskatchewan 88% of operators were certified, while in Manitoba the figure was 74% –both in sight of the 100% target.

The Standing Senate Committee Final Report on Safe Drinking Water for First Nations expressed the opinion that training and certification efforts have not lived up to expectations:

... the Committee is alarmed at how little progress has been made in training and certifying First Nations water systems operators. Even the best facilities cannot ensure safe water unless properly managed and operated. Although INAC is working to strengthen and expand its Circuit Rider Training Program–a program designed to enhance First Nations capacity to operate and maintain water system and wastewater systems–these efforts are clearly insufficient...The Committee feels strongly that a comprehensive, long-term training program must be put in place immediately.[NOTE 21]

The Expert Panel offered some explanations for the difficulties:

• This problem is not specific to First Nations communities. "In British Columbia, Manitoba and Ontario, as instances, it would appear that the proportion of small and rural communities served by fully accredited operators is about as high as it is in non-Aboriginal communities. All communities are currently chasing a qualified labour force that is too small and that, given the time it takes to train and certify people, cannot be expanded overnight. Both groups have some distance to go to reach provincial standards, but the situation is far from hopeless. [NOTE 22]
  
  
• It is also generally acknowledged that small communities might have difficulty retaining their qualified and certified operators, especially in locations where funding for O&M and compensation for operators are not adequate.

In his testimony before the Standing Senate Committee on Aboriginal Affairs, the Director of Housing for the Assembly of First Nations stated that approximately 25 percent of plant operators in First Nations communities had left for jobs with other municipalities or private companies. [NOTE 23] He says this is because the federal government does not provide adequate funds to keep operators in First Nations communities.

The following tables summarize information on operator certification.

Table 18: Certified Operators by Certification Level

Table 19: Water System Operators Training and Certification

Table 20: Wastewater System Operators Training and Certification

Oversight

An alternative to having all First Nations community systems operated by certified operators is to provide oversight of water systems by certified service providers. This approach was called for in the 2006 Plan of Action. Oversight operators can assist when emergencies arise and can also help train onsite operators.

A 2007 departmental progress report on the implementation of the Plan of Action stated that oversight was being implemented in two phases: a 24-hour emergency line for technical support operational since December 2006, and direct monitoring and review of local operators' logs and records. In some INAC regions, local circuit rider trainers and First Nations technical organizations have taken on this responsibility and begun to implement the second phase.

The highest level of oversight is being implemented through the Safe Water Operations Program, originally developed in Ontario and now available in all regions, with funding from INAC's Capital Facilities Maintenance Program. The Safe Water Operations Program includes full-time, onsite supervision by a certified operator. This will not only help ensure proper operation of facilities, but also support training of local operators.

Improved and consistent water monitoring program delivery

HC defined the outcome of "improved and consistent water monitoring program delivery" as developing the capacity to implement a monitoring program which meets the recommendations of the GCDWQ and the Procedure Manual. The RMAF indicator for this outcome was the assessment of regional program delivery. Given the definition of this immediate outcome, two additional performance indicators were examined to assess whether HC has developed the capacity to meet the logistical requirements needed to adhere to the GCDWQ and the Procedure Manual. These indicators are: (1) the ratio of EHOs to communities, and (2) the percentage of communities with access to CBWMs.

Assessment of Regional Program Delivery

Table 21 indicates that regions report a varying capacity to achieve work-plan objectives. It is difficult to assess the meaning of this self-reported indicator as work-plan objectives may vary by region and the incomplete objectives were not recorded. The remaining 13 indicators described in this report offer a more concrete picture of program delivery.

Table 21: Percentage of Regional Work Plan Objectives as per the FNWMS Met by Fiscal Year as Reported in the REHM Survey

Ratio of EHOs to Communities

The indicator chosen to measure the coverage provided by EHOs was the ratio of EHOs to communities. In 2005-2006, the EHO questionnaire identified a total of 682 First Nations communities with at least one water system (i.e., community water system, trucked water system or public water system) that required monitoring under the FNWMS. This total was higher than the previous year's (N = 613), supposedly due to improved reporting practices. As a result, the 2005-2006 value (N = 682) was used to calculate the EHOs:community ratio for both years.

The ratio of EHOs:community increased in most regions. Furthermore, in five of the seven regions, the target value (as determined in 2002) was attained (see Table 22). Only the Ontario and Pacific regions had yet to reach their target ratio.

Table 22: The Number of EHO per one community by Region and Year Compared to the Target Values

Note: The gap analysis completed by HC in 2002 identified regional targets. The targets were based on the amount of time required by EHOs to travel to communities, monitor drinking-water quality, train CBWMs and maintain QA/QC programs.

Percentage of Community Sites with Access to Trained CBWMs

The EHO Questionnaire asked respondents if at any point during the previous fiscal year there were "systems in the community monitored by a trained CBWM for bacteria". These data were then used to assess the number and percentage of community sites with "access" to CBWMs.

The percentage of community sites with access to CBWMs increased from 72 percent in 2004-2005 to 87 percent in 2005-2006 (see Table 23). The goal is 100 percent. Regionally there was some variation with slightly lower than average percentages in Ontario and Pacific regions, the regions with the largest number of community sites to cover (Table 23).

One of the goals of the FNWMS was to increase the number of CBWMs. Many First Nations communities are in remote locations and lack CBWMs, making it difficult for EHOs to meet weekly sampling requirements. More CBWMs means that problems can be detected more quickly and cost-effectively. In 2002, the percentage of community sites reporting access to CBWMs varied regionally from 21 to 84 percent; the national average was 40 percent. The national average rose to 87 percent in 2005, a clear indication that the Strategy is training more CBWMs as planned.

Table 23: The Number and Percentage of Community Sites Reporting Access to a CBWM by Fiscal Year

Note: The total number of communities was taken from the 2005-2006 EHO Questionnaire

Increase in frequency of monitoring relative to GCDWQ

The FNWMS RMAF lists two performance indicators for sampling frequency: (1) the number of samples collected, tested, interpreted and communicated back to First Nations and other relevant stakeholders, and (2) the frequency of samples collected, tested, interpreted and communicated back to First Nations and other relevant stakeholders.

In general, all test results are communicated to First Nations and other relevant stakeholders. When a sample exceeds guidelines, communities are notified by email messages issued automatically by the early warning database or directly by a CBWM or EHO. Communities that have access to a regional water-monitoring database (e.g., Water Trax, Eau-Water) have continuous access to all relevant test results. The existence of these communications channels means that the EHO Questionnaire need not include questions about how test results are disseminated. Performance-indicator information gathered via the Questionnaire are: (1) the number of samples analysed and interpreted for bacterial contaminants, and (2) the frequency of samples collected, tested and interpreted for bacterial, chemical and radiological contaminants. The EHO Questionnaire asks only whether chemical and radiological samples were collected at least once during the year and does not ask how many of such samples were collected.

An additional indicator measured the number of communities that employ the recommended analytical quality-control procedure. Increased frequency of sampling is unlikely to promote safe drinking water if test results are unreliable. To obtain valid results, a quality assurance (QA) framework and quality-control techniques should be used. One aspect of the QA framework is quality control (QC), which assesses the reliability of the analytical techniques being employed. Increased sampling, especially if conducted onsite by CBWM, must be accompanied by increased QC analysis.

The RMAF originally contained the indicator "the number of samples collected by CBWMs that do not pass quality assurance/quality control". Data cannot be collected on this indicator as worded. QC aims to detect systemic problems with sampling and analytical techniques; it does not assess whether individual samples "pass QA/QC". Consequently, the indicator was replaced by "the number of water systems for which monitoring included the recommended quality control." The Procedure Manual recommends that at least 10 percent of all samples analysed in the community also be sent to an external, accredited lab for verification. As a condition of accreditation, these laboratories must maintain adequate quality-control systems.

Number of Samples Analysed and Interpreted

The total number of samples taken and analysed for bacterial contaminants by: (a) an accredited external laboratory, and (b) onsite analysts (e.g. EHOs and CBWMs) using portable lab kits were calculated for all water systems. Values for 2002 were taken from the gap analysis conducted by HC during the design stage of the FNWMS, and do not distinguish lab testing from onsite kits. The data for 2004-2005 and 2005-2006 were drawn from EHO Questionnaires. Data for 2006-2007 are preliminary estimates provided by the REHMs.

Between 2002 and 2006-2007, the number of samples analysed across Canada increased by 700 percent (Figure 1, Table 24). The number of onsite samples increased more than lab samples (Figure 1), as more portable kits were provided to communities and more CBWMs were trained to use them. The distribution of inexpensive, easy-to-use onsite sampling kits was a key part of HC's strategy to increase monitoring frequency in First Nations communities. For remote communities, onsite sampling is often the only method to test water for bacterial contaminants, as analysis must be done within 24 hours of sampling.

Most of the increase in sampling occurred between 2002 and 2004-2005, and an unknown portion of the change may be attributed to differences in data-collection techniques (Table 24). Between 2004-2005 and 2006-2007, when similar data-collection methods were employed, the number of samples increased 57 percent nationally (Table 24). The largest increase during this period was recorded in Saskatchewan, although the rest of Canada showed a 21 percent increase in number of samples taken. Despite possible differences in data-collection methods, it seems probable that the number of samples collected increased dramatically between 2002 and 2007.

Figure 1. Number of bacteriological samples analyzed in an accredited, external lab and onsite using portable test kids by fiscal year. Data for 2002 are for lab and onsite samples combined.

Table 24: The Number of Bacterial Samples Analysed by a Lab and Onsite by Region and Fiscal Year

Note: 2006-2007 data are preliminary values provided by the REHMs, but should be similar to values which will be reported in the EHO Questionnaire as both EHOs and REHMs have access to the same data sources.

Frequency of Bacteriological Samples Collected, Tested and Interpreted

HC's 2002 gap assessment indicated that drinking-water quality monitoring in First Nations communities was insufficient to protect human health. For example, the monitoring of bacteriological contaminants in distribution systems only met, on average, 29% of the frequency recommended in the GCDWQ. The Procedure Manual for Safe Drinking Water in First Nations Communities South of 60 (the Procedure Manual) describes appropriate practices for monitoring water quality in distribution systems with five or more connections and in cisterns, including sampling and testing frequencies, as per the GCDWQ and From Source to Tap: the Multi-Barrier Approach to Safe Drinking Water.

According to the GCDWQ and the Procedure Manual, the recommended sampling frequencies for bacteria in the three water distribution systems are:

• for Community Water Systems (CWSs), a minimum for four times per month or weekly,
  
  
• for Public Water Systems (PWSs), four times per year (quarterly), and
  
  
• for Trucked Water Systems (TWSs), four times per year (quarterly).

The recommended frequencies for chemical and radiological sampling are once per year, with one exception: tests for Trihalomethane (THM), a disinfection by-product, must be conducted four times per year, once each season.

For the purpose of this evaluation, the percentage of water systems that met the required sampling frequency as per the Procedure Manual was used as a performance indicator. The annual EHO Questionnaire captures data on the actual bacterial, chemical and radiological sampling frequencies. Data on the frequency of bacterial testing are reported using a five-point rating scale of weekly, biweekly, monthly, sporadically and never. The chemical and THM sampling data was reported on a dyadic scale of yes and no, with a yes response indicating that sampling was done.

In fiscal 2005-2006, 98 percent of CWSs were monitored to some degree for bacteriological parameters. Nationally, 43 percent of CWSs were monitored weekly for bacteria as required (see Figure 2). An additional 41 percent were monitored on a biweekly or monthly basis. The remaining 14 percent of CWSs were monitored occasionally. While the Procedure Manual provides guidelines on the frequency of sampling for CWSs as well as on where to collect samples (i.e., at the water supply or in the distribution system), unfortunately information was not collected on the part(s) of the system where the sample was taken.

For PWSs, in 2005-2006, 91 percent were monitored to some degree. The percentage of PWSs that met required sampling frequency was 49 percent (see Figure 2). This percentage may underestimate the actual frequency, however, as 42 percent of PWSs were reported as having "sporadic" testing, which may include systems that were tested four times a year, as required. In addition, as can be seen in Figure 3, there was considerable regional variation.

Sufficient information for analysis on the frequency of testing in TWSs was not available at the time of this evaluation. The method to collect and interpret data from this type of system is still being developed. In addition, due to the large number of cisterns in poor condition and therefore, not being used to store water, many TWSs are not tested.

Compliance with the guidelines for chemical testing was highest in CWSs, as 77 percent of all systems were tested annually. Compliance was considerably lower in PWSs at 47 percent (see Figure 4). The same pattern was observed for radiological testing, but with much lower compliance rates. It should be noted that for chemical and radiological parameters, EHOs were asked only whether samples had been collected, and not about how often they were collected or which parameters were tested. As the recommended frequency for chemical and radiological parameters is once per year, it was assumed that any system that was sampled met the recommended frequency. EHOs were asked specifically if samples were tested for THM. As can be seen in Figure 4, 54 percent of CWSs and 25 percent of PWSs were sampled for THM in 2005-2006. However, it was not possible to determine if they met the quarterly sampling frequency recommended by the Procedure Manual. Information on THM sampling frequency will be gathered in the 2006-2007 EHO Questionnaire.

Variation in the percentage of systems sampled between 2004-2005 and 2005-2006 was not highlighted in this report and trends should be interpreted cautiously. The absolute number of water systems meeting the sampling targets for all parameters increased in 2005-2006. However, in some instances the total number of water systems also increased. The data for 2006-2007 should provide a clearer illustration of trends in sampling frequency, as the set of water systems will be similar to that of 2005-2006.

Assessing the proportion of distribution systems which met recommended sampling frequencies was complicated by limitations in the EHO Questionnaire. Unfortunately, the five-point rating scale for bacterial sampling did not allow for the identification of systems tested "quarterly", the recommended frequency for PWSs and TWSs. Furthermore, the EHO Questionnaire does not provide a complete assessment of sampling, as it failed to adequately identify sampling location. The frequency of sampling reported does not distinguish between samples taken from the water sources (e.g. water-treatment plant) and from distribution systems. For CWSs, for example, samples should be collected weekly at the point where the water enters the distribution system, and from at least two points within the distribution system. Conversely, the EHO Questionnaire does not capture information about sampling conducted outside the community, such as when water is provided by a neighbouring municipality. In these cases, the EHO Questionnaire may underestimate actual sampling frequency

Another limitation of the EHO Questionnaire is that for TWSs, it does not identify sampling frequencies for individual cisterns; rather it identifies only sampling frequencies for entire distribution systems. Reported frequencies, therefore, may refer to samples taken from the central supply, trucks or cisterns.

Figure 2. Percentage of Community Water Systems (CWSs) and Public Water Systems (PWSs) Employing the Required Bacterial Sampling Frequency for fiscal 2004-2005 and 2005-2006.

Figure 3. Regional Distribution in Percentage of Water Distribution Systems with Required Bacterial Sampling Frequencies in Fiscal Year 2005-2006 by Water System Type.

Figure 4. Percentage of CWS and PWS Employing the Required Sampling Frequencies for Chemical, THM and Radiological Parameters by Each Type of Water System

Number of Water Systems Where Monitoring Included Recommended Quality Control

The number of water systems where monitoring included the recommended quality control was calculated by determining the percentage of water systems where: (a) the EHO answered yes to "Were at least 10% of all samples collected by the CBWM sent to an accredited laboratory for QA/QC purposes" or (b) the EHO answered no to the above question but reported that all bacterial samples were analysed by an external laboratory. The target for the number of water systems where monitoring included the recommended quality control would ideally be 100%. It must be acknowledged, however, that some communities are too remote to deliver samples to accredited labs within the recommended 24-hour time period.

Nationally, less than half of all water systems reported employing the recommended analytical QC procedure, and there was considerable regional variation (see Figure 5). Again, year-over-year variations should be interpreted cautiously due to changes in the set of water systems reported. The drop observed in this indicator for Québec Region in 2005-2006, for example, was due to the addition of 12 public water systems in a community for which no QC was conducted, thereby increasing this single community's influence on the regional average. The public systems had not been included in the previous year's survey. Overall, there was no apparent national trend in the percentage of communities employing QC, and most regions remained well below the target of 100 percent.

Figure 5 Percentage of Water Distribution Employing an Analytical QC Procedure of Having at least 10% of Samples Tested through a Lab by Region and Fiscal Year.

It should be noted that several quality-control procedures are currently being employed. A few examples include CBWM training, distribution of procedure manuals, use of accredited laboratories, and sending 10 percent of onsite samples to accredited labs. However, no standardized QA system and compliance-assessment tool is consistently employed in all regions.

Intermediate outcomes

The FNWMS aims to achieve the following intermediate outcomes:

• water and wastewater facilities meet established federal standards and guidelines ederal standards and guidelines
  
  
• increased ability to identify drinking-water quality problems and potential waterborne illness

This section analyzes the Strategy's progress against these outcomes.

Water and Wastewater facilities meet established federal standards and guideline

According to the Program's RMAF, the indicator defined to assess this outcome is the "number of water and wastewater facilities that are at low-, medium-, and high-risk". Progress is measured against the baseline numbers for low-, medium-, and high-risk systems recorded in the national assessment of all water and wastewater systems undertaken by INAC in 2002/03.

The Risk Scale

The evaluation of risk for a drinking-water system considered five factors: water source, system design, operations, reporting practices, and operator training and certification. Risk factors for wastewater-treatment systems are the same as those for drinking water except for the first factor, which is "effluent receiver". For each of these factors a score is determined on a scale of 1 to 10. A high-risk system requires a score of eight points or higher; a medium-risk system, a score of five, six or seven; and a low-risk system a score of four or less.

In 2006, INAC introduced a weighting method to calculate overall risk levels. Factors related to system design and operations each account for 30 percent of the overall score for a drinking water system; operator training and certification account for 20 percent; while water source and reporting practices each account for 10 percent. With respect to wastewater systems, system design and operations each account for 25 percent of the total score, operators account for 20 percent, and water source and reporting practices each account for 10 percent.

In 2006, INAC defined risk in the following manner:

• Low-risk: system has minor deficiencies and meets the quality parameters specified by the appropriate Canadian guidelines (typically the GCDWQ and the 1976 Effluent Guidelines for wastewater effluent).
  
  
• Medium-risk: system has one or more deficiencies and poses a moderate risk to water quality and human health. The system would not typically require immediate action, but action is needed to avoid future problems.
  
  
• High-risk: system has one or more major deficiencies, poses a significant risk to water quality and may cause health, safety and environmental concerns. Public alerts, such as boil-water advisories, may be required. May also result in water quality advisories against drinking the water, repetitive non-compliance with guidelines, and inadequate water supply. Regional officials, in collaboration with First Nations, are required to take immediate corrective action.

Guidelines and Risk Levels

The measurement of this indicator raises the question of the link between "established federal guidelines" and risk levels as defined above. While the expected outcome is that water and wastewater systems meet established guidelines, systems' performance are reported only in terms of their risk level, not in terms of compliance with guidelines.

There is no explicit relationship between risk levels and compliance. It is not clear at what point of the 10 point risk assessment scale a system ceases to meet established federal standards and guidelines. Based on the defined risk levels, it can be inferred that a high-risk system is "at risk" of not meeting established standards and guidelines, rather than failing to meet them. In other words, the risk assessment categories and compliance with existing guidelines are measuring different things. One cannot be used as a substitute for the other as suggested by Strategy's accountability and performance framework, specifically its performance indicators.

Variance to Baseline

According to INAC's 2003 assessment of First Nations' water systems, 29 percent or 218 of 740, of all systems were high-risk; 337 (46 percent) were medium-risk, and 185 (25 percent) were low-risk. The same assessment found that 74 of 462 (16 percent) of all wastewater systems examined were high-risk; 201 (44 percent) were medium-risk; and, 187 (40 percent) were low-risk. This information provided a baseline to measure the Strategy's performance.

In July 2007, INAC's WATERS database reported that 13.2 percent of all water systems (98) [NOTE 24] were high-risk, 48 percent (355) were medium-risk, and 38.8 percent (288) were low-risk. Among wastewater systems, 14 percent (69) were high-risk, 39 percent (191) medium-risk and 42 percent (207) low-risk.

Table 25: Risk levels of water and wastewater systems at baseline and in July 2007

Between 2003 and 2007, the percentage of high-risk systems declined from 23 to 13 percent, yet the overall objective of the FNWMS, i.e. the elimination of all high-risk systems by 2006, was not achieved.

The numbers cited above must be interpreted cautiously, given that the original assessments may have been conducted inconsistently from region to region. Furthermore, recent assessments have used new tools, including a weighted scale and detailed protocol. Any assessment of the Strategy's performance must also consider the large number of systems currently rated medium-risk.

Trends Analyses

The numbers cited above suggest an overall reduction in the percentage of high-risk systems, little change in the percentage of systems that pose a medium-risk, and an increase in the percentage of low-risk systems equal to the reduction in the percentage of high-risk systems.

Evaluators further analyzed INAC's WATERS database to better understand these trends in the Strategy's performance but found significant inconsistencies in relevant data. The number of systems included in WATERS has changed over time, complicating year-to-year comparisons. Furthermore, database maintenance undertaken by the Capital Facilities Maintenance Program revealed that the 2002 baseline may have included erroneous information.

Evaluators subsequently selected a set of records on specific water systems for review. These included systems with risk ratings since 20002/03 and excluded all subsequently archived systems. Systems built since 2003/04 were included. The review examined 175 high-risk systems (as opposed to the 218 identified in the 2003 national assessment), 296 medium-risk systems and 184 low-risk systems. An analysis of this subset revealed the following trends:

• Of the initial 175 systems ranked as high-risk, 41 were now ranked as low-risk and 69 as medium-risk; as of July 2007, 64 systems continued to pose a high-risk.
  
  
• The overall proportion of low-risk systems increased by 10 percent in the Strategy's first year (and has remained relatively constant at 38 percent of all systems). In this first year of the Strategy, the actual number of low-risk systems rose from 184 to 255, an increase of 71 systems. During this same period, the number of high and medium-risk systems declined by 21. While the increase in the number of low-risk systems (71) can be attributed in part to the reduction in the number of high and medium-risk systems (21), and in part to the inclusion of new systems and the use of new measurements, it is not clear that these results can be attributed to the Strategy.
  
  
• Of the 288 systems rated as low-risk in 2006-2007, 41 (14 percent) were previously ranked as high-risk; 79 (27 percent) medium-risk, and 143 (50 percent) remain unchanged.
  
  
• From 2003/04 to 2006/07, the absolute number and proportion of medium-risk systems within the files reviewed increased from 296 (45 percent) to 355 (48 percent). Of the initial 296 medium-risk systems, 216 remained at that level in 2006/07.

Table 26: Systems (with a risk rating in 2002/03 that were not archived) by risk levels per year

Table 27: Changes in systems first rated as high-risk within the subset 2002-2003

The March 2006 Plan of Action introduced another baseline measurement: INAC reported that 193 [NOTE 25] high-risk systems existed in 170 First Nations communities

Increased ability to identify drinking-water quality problems and potential waterborne diseases

The FNWMS RMAF lists two indicators for this intermediate outcome: (1) the number of waterborne disease outbreaks identified, and (2) the number of water-quality problems identified.

One of the objectives of the Strategy was to develop an Early Warning Database that provides test results in a timely manner to all stakeholders as described in the Procedure Manual. Such a database would enable stakeholders to coordinate their responses to potential problems and make timely, informed decisions. Therefore, access to an Early Warning System has been included as another measure for the outcome of "increased ability to identify drinking-water quality problems and potential waterborne diseases".

Number of Waterborne Disease Outbreaks Identified

According to the REHM Survey no instance of an outbreak of a possible waterborne gastrointestinal illness was reported, in 2005-2006 and 2004-2005.

Although HC does not currently maintain a surveillance system for collecting and periodically reporting data on the occurrence and cause of waterborne-disease outbreaks, the EHOs provide this information annually via the EHO Questionnaire. To address this gap, HC has undertaken a few initiatives.

In 2006-2007, HC identified current procedures to address waterborne threats to human health in First Nations communities, as well as the roles and responsibilities of nurses in the surveillance of waterborne illnesses in First Nations communities. This information will inform draft procedures to address waterborne threats to human health in First Nations communities south of 60.

In a second initiative, HC is collaborating with Public Health Agency of Canada (PHAC) in the development of C-EnterNet, a tool that facilitates comprehensive site surveillance for enteric disease by local public-health units. C-EnterNet includes simultaneous and in-depth community-based investigation of foodborne and waterborne diseases and exposure. The goal of this program is to better understand the true burden of enteric disease in Canada, to assess the value of intervention programs and to accurately and effectively inform policy on food and water safety. The applicability of this model as a surveillance tool for waterborne enteric diseases in First Nations communities is being explored. The C-EnterNet model is based on the Centers for Disease Control and Prevention (CDC) FoodNet sentinel site model - a leading-edge surveillance approach implemented to reduce the occurrence and impact of foodborne diseases in the United States.

Number of Water Quality Problems Identified

Drinking water advisories (DWAs) are common in small and remote communities across Canada. In general, DWAs are issued when water quality standards are not met, or when a risk to public health has been detected. In most cases, the advisory is precautionary and indicates that a risk has been detected, not that an actual hazard (i.e., pathogen) is present. Therefore, the issuance of a DWA indicates that a monitoring program has recognized and responded to a potential health risk.

From a public-health standpoint an appropriate target for a water monitoring program is the issuance of a DWA whenever monitoring shows that drinking water poses a risk to human health. By increasing water-quality monitoring and reporting, it is more likely that problems will be detected, which in turn drives up the number of DWAs issued. It should be noted that other factors, such as severe weather events, can also impact water quality and the number of DWAs. The absolute number of DWAs issued during a given period must be interpreted within this context.

HC regional offices provide weekly reports on DWAs to Headquarters. DWAs are comprised of the number of Boil Water Advisories, Boil Water Orders and Do Not Consume directives. In 2004-2005, there were 92 DWA issued and 78 carried over from previous years. In 2005-2006, there were 100 DWA issued and 105 carried over from previous years. In 2005-2006, a large proportion of the new DWAs were short in duration, while a small number continued for a longer period of time (see Table 28).

Some First Nations communities are on long-term DWAs and the reasons for this are often complex. Many First Nations communities face the same challenges in providing safe drinking water as do other small, remote or isolated communities, such as difficulties in finding and retaining qualified water treatment plant operators. Other reasons for long-term DWAs may be the decision of a community to lower or turn off the chlorinator because of community concerns about the taste of chlorine in drinking water after treatment and/or the time required to perform upgrades or replacements to a water facility in a community. To further understand the reasons for DWAs within First Nations communities, HC National Headquarters is in the process of gaining full access to regional water databases and future analysis will include a breakdown of DWA by cause.

Table 28: DWA by status and year

Note: This analysis is based on best available data from all the regions.

Early Warning Databases

Early Warning Databases currently operate in all regions excluding Saskatchewan. By comparison, in 2002 only two out of seven regions had access to early warning databases. These centralized databases record sampling and test results, and automatically send out notifications of exceedances. Early Warning Databases provide EHOs with rapid access to test results and enable authorized personnel to generate reports. Early Warning Databases also provide communities and other appropriate stakeholders' access to drinking water results.

Québec region uses the federal database known as Eau-Water, while Pacific, Alberta, Manitoba, Ontario and Atlantic regions use a private database called WaterTrax. Saskatchewan is in negotiation to secure an appropriate database and, in the meantime, collects data using EXCEL spreadsheets.

It is expected with the transmission of monitoring results through the Early Warning Databases that timely and appropriate action is taken by First Nations communities in response to the identification of hazards in the water. However, the Early Warning Databases are not equipped to track the amount of time that lapses between the reporting of a result and the action taken. Furthermore, the databases do not capture information on the actual action taken to assess the appropriateness of the response.

Final Outcomes

The RMAF identifies three final outcomes for the Strategy:

• Reduced public health risk associated with water quality and supply issues
  
  
• Increased capacity of First Nations communities to address potential water quality problems, and
  
  
• Increase the confidence of First Nations in the quality of their drinking water.

Reduced health risk associated with water quality and supplies issues

Reducing the potential health risks posed by poor-quality drinking water requires a multiple-barrier approach that includes source protection, water treatment and monitoring. Monitoring should take place at source, within the distribution system and at tap, only the last of these falls under the jurisdiction of HC. The FNWMS RMAF performance indicator for this outcome is the number of confirmed outbreaks of waterborne diseases.

No waterborne gastro-intestinal illnesses have been confirmed in First Nations communities since 2002.

The assessment of this final outcome requires performance indicators to assess how the FNWMS reduces health risk associated with water quality. The indicator discussed above (i.e., the number of confirmed waterborne disease cases and outbreaks), while being an outcome of interest, it is a poor indicator of change in health risk related to improvements in the monitoring program. The absence of illness does not necessarily indicate reduced health risk attributable to the monitoring program. Firstly, there were few, if any, confirmed waterborne illnesses in First Nations communities prior to the establishment of the FNWMS. This may be due to the difficulty of detecting and confirming drinking-water related gastrointestinal illness. Many cases of gastrointestinal illness go unreported, as many people do not seek medical attention unless the symptoms are severe. Even for those who seek medical attention, the causal agent is rarely identified, preventing confirmation of a waterborne illness. Secondly, if water treatment is functioning properly it is unlikely there will be any waterborne illnesses in the community, regardless of the quality of monitoring.

The ability of the Strategy to monitor First Nations communities' drinking water has been examined by assessing the delivery infrastructure of the monitoring program and the frequency and quality of sampling, which encompass the immediate and intermediate outcomes previously described. However, change in health risk related to improvements in the monitoring program would require an examination of the implementation of effective monitoring programs within communities and their response time when water quality problems are identified. Therefore, two performance indicators could be: (1) the number of communities meeting all requirements for an effective drinking-water quality monitoring program, and (2) the timeliness and appropriateness of the response taken when monitoring detected a water quality problem. For the first indicator, each community would need to be scored on a pass or fail basis using criteria related to the type, frequency and quality of sampling. The second indicator could measure the time between confirmation of a problem result (e.g., positive E. coli test) and implementation of the appropriate response (e.g., setting of a drinking water advisory).

Increase capacity of First Nations communities to address potential water quality problems

The performance indicator in the FNWMS RMAF for this outcome is the number of First Nations communities with trained operators, CBWMs and water-quality testing and sampling kits.

A key objective of the FNWMS was to increase the capacity of First Nations communities to monitor drinking water. This was done by training CBWMs and providing them with portable lab kits to test for bacterial contaminants. The number of communities with trained CBWMs and access to portable lab kits has increased steadily since the FNWMS was initiated in 2002 (see Table 29). In 2005-2006, 86 percent of communities had access to trained CBWMs and 91 percent had access to kits for onsite bacteriological analysis of drinking water. These figures indicate that the FNWMS is close to attaining one of its goals–that all First Nations communities have access to CBWMs and lab kits.

Table 29: Number (N) and percentage (%) of communities with community based water monitors (CBWM) and portable kits for onsite bacteriological analysis of drinking water.

Note Total number of communities is 682.

Increased First Nations communities confidence in their drinking water

The performance indicator outlined in the FNWMS RMAF as a measure of this final outcome is the perception of First Nations community residents that the quality of drinking water has improved.

In 2007, HC conducted public-opinion research among members of First Nations communities.

Results of the research suggest that currently:

• 27 percent of respondents consider their tap water to be very safe,
  
  
• 35 percent believe their tap water is somewhat safe,
  
  
• 20 percent believe their tap water is somewhat unsafe,
  
  
• 16 percent think their tap water is unsafe, and
  
  
• 2 percent did not respond.

Relative to five years ago:

• 41 percent of respondents believe their tap water is currently better,
  
  
• 23 percent consider their tap water to be currently worse,
  
  
• 33 percent believe their tap water has not changed in the last five years, and
  
  
• 3 percent did not respond.

The majority of First Nations residents surveyed rated their drinking-water quality as good or moderate (66 percent) and safe for consumption (62 percent). Nearly a third rated their drinking water as poor and unsafe for consumption. When asked how water quality has changed over time, 41 percent believe it had improved over the past five years, while 32 percent said there was no change. Nearly a quarter of respondents believed that water quality had declined over the past five years.

Perceptions of water quality and safety varied significantly in urban and rural settings. Urban residents were more likely than rural residents to rate their water as good and safe for consumption. Residents of the British Columbia, Québec and Atlantic regions were also more positive about water quality than residents of the Alberta, Manitoba and Ontario regions. Still, the majority of residents in all locations rated their drinking water as good or moderate, and safe to consume. The urban versus rural difference may, in part, be the result of how water is delivered. Respondents with piped water systems were more positive about water quality and safety than those who drew water from cisterns or wells, a practice more common in rural communities.

Value for Money

This section presents evaluation findings on value for money for investments in the upgrading of water systems (INAC) as well as for the monitoring of drinking-water quality (HC).

INAC - Upgrades to Water Systems

The FNWMS RMAF did not include indicators or measures to assess cost-effectiveness. Cost-effectiveness related to capital costs for water and wastewater systems is normally measured in terms of cost per service connection, per person served, and per mega litre of water treated. Program databases do not include the information necessary to measure cost per service connection and per person served. Some relevant information does exist, however, and this section attempts to link the Strategy's costs to its outcomes.

Overall assessment

The Strategy committed to eliminate high-risk systems through the investment of an additional $600 million in First Nations water and wastewater systems. Although there has been considerable progress, 98 high and 355 medium-risk systems remain.

Since the Strategy's introduction, a number of observations have been made with respect to its cost-effectiveness, including a 2005 report of the Commissioner of the Environment and Sustainable Development (CESD) which observed that:

Despite the... federal funds invested, a significant proportion of drinking water systems in First Nations communities continue to deliver drinking water whose quality or safety is at risk. Although access to drinking water has improved, the design, construction, operation, and maintenance of many water systems are still deficient. [NOTE 26]

The CESD report also includes anecdotal information about two cases (of 28 reviewed) where decisions were made that affected cost-effectiveness. In one case, INAC approved the most expensive design option for a water system, adding $630,000 to the estimated construction costs–36 percent more than the lowest-cost option. A case study provides another example where an expenditure of $782,000 yielded extremely poor results; alternative and innovative options had not been considered. [NOTE 27]

Information compiled in 2006 by an INAC regional employee summarizes and compares the costs of major water-servicing projects in remote communities with small to medium populations in a similar geographic area (Northern Ontario). Average total water construction costs were $12.1 million and the average cost per connection was $66,300. The lowest cost per connection was $44,500, while the highest was $111,600.

To accurately measure cost-effectiveness of a project requires comparisons to projects of similar nature and size. Such information on the Strategy was not available to the evaluators. A study commissioned by INAC established that the capital cost of central water supply and wastewater infrastructure in urban areas is in the range of $25,000 per house connection. The study found that costs increase dramatically outside urban areas:

Remote and isolated community costs will be significantly higher, up to 3–4 times the construction cost in the south. On top of that, the low density of development may mean that 3–4 times the piping length (both sewer and water) must be installed compared to compact, urban areas. For example, in a typical urban community, the length of piping to service the population would be about 1–2 m per person. This represents a capital cost of approximately $1,800 per person (i.e. $900 / m for water and sewer) or $9,000 per household for a 5-person family. In a rural or spread out village, the separation of homes may require as much as 8 m of piping per person, resulting in an additional cost of $36,000 per home. Factoring in the construction cost for remote, isolated communities on rock could easily yield a cost of $144,000 per connection.[NOTE 28]

Given these realities, the cost estimates for major water-servicing projects in First Nations communities in Northern Ontario–although high–are not surprising.

According to the same study, another factor to consider when assessing the cost-effectiveness of the Strategy is the level of servicing.

Much of small non-aboriginal communities in Canada rely upon private onsite services to meet their water and wastewater needs. Communal systems are also typically used where it is considered impractical to construct a large centralized system due to the high cost and/or low density of population. According to Environment Canada's Municipal Utility Database (MUD) survey in 2001, more than 3 million rural homes and buildings are not connected to municipal systems. Dalhousie University has reported that more than 50% of the population of Nova Scotia depends upon onsite sewage systems, principally onsite sewage disposal systems and leaching beds. There are 26 villages within the newly amalgamated City of Ottawa, and most of these are reliant upon onsite or communal facilities to meet their needs; mostly private wells and septic systems. [NOTE 29]

INAC does not fund the construction, operation and maintenance of private wells and septic tanks. As a result, the Strategy essentially excluded an option that is considered to be cost-effective in communities with needs similar to some, if not many, First Nations communities. Evaluators were unable to find a written rationale for the exclusion of private wells. It has been suggested that wells and septic beds are typically the responsibility of homeowners. As such, they could be considered a housing issue or an individual responsibility, an approach rejected by AFN in their presentation to the Standing Senate Committee on Aboriginal Affairs.

Cost Per Outcome in Risk Reduction

Evaluators also attempted to measure costs associated with the Strategy's outcomes. A cost per point of reduction in overall risk rating might have provided useful information had total construction costs been readily available, but this was not the case.

Evaluators also tried to determine which of the Strategy's investments were the most cost-effective. Using the 175 high-risk systems reviewed, the estimated cost-per-point of risk reduction was compared with capital and other expenses (such as operations and maintenance). It was assumed that capital expenses were directed at reducing risks associated with water sources and system designs, while the operations and maintenance, and training expenses targeted reductions associated with operations, reporting and operators. This assessment shows that, within the subset of systems, the average cost-per-point reduction for capital expenses was $1.5M and $70,000 for operations and maintenance and training expenses.

Evaluators also tried to determine each risk factor's contribution to overall risk loads of the systems. A simple comparison of the average score per risk category revealed that for high-risk systems, operations were the most significant factor. For low-risk systems, the source of water was the most important factor. In a multi-barrier approach, one can assume that the entire system is built and operated to mitigate the risk posed by the water source. Therefore, it is normal that the source of water source be the highest rated in a well-functioning system.

These calculations, although rudimentary, clearly show that investments in operations and maintenance are far more cost-effective than investments in capital.

Table 30: Average score per risk factor per risk level

Cost Drivers

The following factors affect the cost of building and operating water and wastewater systems in First Nations communities:

• The relatively small size and remote location of many First Nations communities serve to increase costs. Communities targeted by the Strategy, to use the words of the Chair of the Expert Panel: "are the hardest to deal with anywhere. This is not a comment on about aboriginality, but about size. Very small towns in Ontario are having difficult problems meeting the standards".
  
  
• Important economies of scale exist for water and wastewater treatment, for both capital, and operations and maintenance. Past a certain point, the cost per connection drops dramatically as the scale of the system increases. The same is true for operations and maintenance costs. It was calculated that the operating costs for municipalities in southern Ontario with populations of more than 40,000 were as low as $386 per ML; operating costs for municipalities with population of less than 5,000 were $1,028 per ML. [NOTE 30]
  
  
• The same study reported that according to the Association of Consulting Engineers of Canada, operations and maintenance of an infrastructure project can account for 80 to 93 percent of lifecycle costs, leading the authors to conclude that high operating costs can be unmanageable for communities with small populations.
  
  
• It is also generally accepted that microbiological exceedance in the water is inversely related to system size.
  
  
• Finally, there is the issue of increases in the cost of construction. Since the inception of the Strategy, construction costs, especially those in western Canada–home to most of the systems targeted by the Strategy–have more than doubled.

HC – Drinking-Water Monitoring

The assessment of the value generated by HC's water monitoring function within the Strategy, considered elements of the Value for Money Profile currently being developed by Treasury Board Secretariat's Centre for Excellence for Evaluation. [NOTE 31] The assessment of value was based on three criteria:

• Economy: appropriate qualities and quantities of resources are obtained at the lowest cost;
  
  
• Efficiency: a given amount of resources produces optimum outputs; and
  
  
• Cost-effectiveness: the unit costs of outcomes are minimized.

Considering these criteria, HC's evidence base did provide a number of challenges in determining the efficiency and cost-effectiveness of HC's DWSP. The draft Value for Money Profile states that to demonstrate efficiency and cost-effectiveness, an evidence base must be available from which to draw conclusions. In this case, a program must have operation plans with well articulated input costs and targets, as well as electronic tracking of costs and data that is validated.

The DWSP is highly decentralized, with regional offices developing individual planning documents independently of HC Headquarters. Furthermore, it appears that planning documents are not routinely shared between regional offices and Headquarters. Thus, it is not possible to determine what targets or objectives regional offices or Headquarters have been setting for the DWSP each year. As a result, the program was unable to demonstrate that it links yearly program expenditures (or input costs) with planned and achieved results (outputs). According to the TBS Guide on Costing Government Services, performance measurement is only valid when there is a comparison between an actual and expected target and any related timeliness, quality and cost measures. Therefore, as a whole, the monitoring component of the Strategy is not generating some of the data needed to fully demonstrate value for money. Nonetheless, this report demonstrates that some outcomes have been achieved and this particular section strives to measure economy on a rudimentary level and provides cost-benefit illustrations.

Economy

Discussions with HC program personnel and management revealed that at the Strategy's inception the water sampling and testing regime was developed based on international best practices and with a view to obtaining the best value for money. Within the pre-existing program structure, it was decided to focus on enhancing community capacity to monitor and test water by extending the Community Based Water Monitoring Program, thus increasing monitoring coverage and reducing the burden of water testing on the limited number of EHOs. In tandem, onsite testing kits would be provided to CBWMs to test for E. Coli and Coliform bacteria. In 2006-2007, Dalhousie University evaluated the Atlantic's region's CBWM program and determined that the program was unique within Canada and internationally, and consistent with international best management practices.

The following table illustrates the difference in annual monitoring costs between the FNWMS approach and an approach using EHOs and accredited labs. Even with training and support costs considered, the FNWMS approach is clearly more cost-effective. More specifically, CBWMs cost less than EHOs and it costs much less to test water with onsite kits than at accredited labs.

Table 31: Cost Comparison between Monitoring Approaches

Note: The average annual salaries of CBWMs and EHOs exclude the costs of benefit programs.

For further illustration of the cost savings of the FNWMS approach, Atlantic regional staff estimated that the use of CBWMs and onsite kits generated annual savings of $442,000 to $553,000 between 2003 and 2007. Additionally, in the Québec region, it costs approximately $46,838 per year for onsite kits and related supplies (bottles and powder reagents); at current prices, it would cost approximately $174,720 per year to test a similar number of samples in accredited laboratories. This is an annual savings of $127,882.

Efficiency

Neither the HC evaluator nor the HC program manager could find appropriate benchmarks for monitoring and testing efficiency to support measurement of the DWSP.

Other attempts to calculate efficiency were not possible due to the data problems previously described: the EHO questionnaire collects limited data on sampling frequency; program plans lack well articulated input costs and targets; and expenditures and data are not accurately tracked and validated.

Cost-effectiveness

Determining the cost-effectiveness of the monitoring component of the Strategy is also challenging. Inherent in the concept of cost-effectiveness is a measure of the cost of a program's outcomes. However, as with many environmental health programs, the DWSP follows a preventative approach –it strives to identify water-quality problems and prevent disease. Various reviewed sources indicate that the costs of outcomes for environmental health programs, such as the DWSP, are difficult to determine.

In a paper on the evaluation of the costs and benefits of water and sanitation improvements at the global level, [NOTE 32] Hutton and Haller note that there are a number of outcomes or benefits which cannot be measured in terms of cost (non-value outcomes/benefits), such as existence value (people value the fact that the environmental good exists) and bequest value (people want future generations to be able to enjoy it). Similarly, during testimony to the Expert Panel on Safe Drinking Water for First Nations, Aboriginal witnesses repeatedly testified to the cultural value of water and the holistic role it plays in their societies. These benefits are not included in Dr. Hutton's analysis quoted below, but could be considered additional outcomes with inherent, non-monetary value. Likewise, other outcomes of the program, such as increasing capacity within First Nations communities to monitor the quality of drinking water, could be considered as benefits with inherent, non-monetary value.

Nonetheless, using a cost-benefit analysis, Dr. Hutton estimated the cost and benefits associated with a variety of improvements to water and sanitation at the global level with results and conclusions focusing on the developing world. Dr. Hutton found that, in developing countries, water and sanitation improvements were cost-beneficial, with a return of between $5 and $28 for each $1 investment in a range of interventions. Using the median score ($14), an investment of $100M in a water program in the developing world yields a cost-benefit of approximately $1.4 B.

Although First Nations communities, much like other small communities across Canada and around the world experience challenges with drinking water supply and management, they do not face the same issues as the developing world. As a result, caution must be used when applying this cost-benefit calculation in North America. However, based on Dr. Hutton's findings, it seems reasonable to attribute some value to the outcomes of an environmental health program, in this case drinking water, through a costing of the benefits of these interventions (such as increased life expectancy, productivity, and quality of life, and reduced morbidity, mortality and health-care expenses).

The potential cost of disease outbreaks provides another compelling illustration. Harrington, Krupnick and Spofford studied a 1983 outbreak in Luzerne County, Pennsylvania that affected 25,000 households, with 370 confirmed cases of Giardiasis. It was estimated that the total cost of the outbreak was $16M - $45M (1984 US dollars). [NOTE 33] This costing accounted for losses to individuals due to medical costs, loss of work, lost productivity and leisure time as well as the actual cost to households of providing uncontaminated water. Costs to government and business were not calculated.

While it may not be possible to evaluate the cost-effectiveness of the water-monitoring component of the Strategy, as defined by Treasury's Board's Value for Money Profile, benefits of disease prevention appear to outweigh monitoring costs.

It should be noted that HC supports the World Health Organization's development of a cost-benefit analysis of delivering safe drinking water to small communities. Where feasible, this work should be applied to the DWSP's reporting and performance measurements.

Relevance of the Strategy

The Strategy's relevance is based on need and its capacity to meet that need. Another measure of its relevance is links to government priorities and goals.

Continuing need and priority

First Nations communities continue to experience inadequate water and wastewater systems. Some 98 drinking-water and 67 wastewater systems are high-risk; with 38 percent of drinking-water systems and 43 percent of wastewater systems on-reserve are of low-risk.

In addition, since the FNWMS implementation, sampling frequencies have improved, and the total number of samples has increased 700 percent nationally. This attention to water monitoring is a direct response to HC's 2002 assessment of the DWSP, which found the frequency of drinking water sampling did not meet the standards set out in the GCDWQ and was insufficient to protect human health. The current increase in sampling also increases the likelihood that risks to public health will be identified early. Furthermore, the increase in the number of EHOs increases the likelihood that appropriate guidance will be provided to Chiefs and Councils when drinking-water problems are identified. Even though these gains have resulted from the Strategy over the last five years, further gains still can be made with respect to the frequency of sampling. Therefore, the continuation of the program is essential if these gains are not to be lost and further gains are to be achieved.

Clean drinking water on-reserves remains a priority of the Government of Canada. Budget 2006 included $450 million to develop innovative new approaches to housing, water and issues facing women, children and families. In March 2006, the Government of Canada launched an action plan to ensure "that residents of First Nations communities enjoy the same protection afforded other Canadians when it comes to drinking water". And most recently, in the October 2007 Speech from the Throne, a commitment was made to ensuring safe drinking water on-reserve. The FNWMS provides the foundation for meeting this commitment.

Appropriateness of the approach

The question, then, is whether continuing the existing approach will provide a solution to the problem. According to the Chair of the Expert Panel, progress to date justifies continuing the approach:

We are now at a stage where the intent of the 1977 policy–standards similar to non-native communities of comparable size and remoteness–seems to be in sight, although good measurement is lacking. ... if the federal government continued its present level of effort for another five years, we should be operating at a level where the number of boil-water advisories is very small and nothing serious to worry about. This is not, in other words, one of those problems in Aboriginal Canada that will persist forever and ever and ever. This is one that can be solved and it can be solved with the application of a good chunk of money for a limited period of time.

The regulatory gap

Regulations have been a major topic of debate. The report by the Commissioner of the Environment and Sustainable Development (CESD) brought the issue to the forefront with the conclusion:

First Nations communities do not benefit from a level of protection comparable to that of people who live off reserves, partly because there are no laws and regulations governing the provision of drinking water in First Nations communities, unlike other communities.

To address this issue, INAC developed the Protocol for Safe Drinking Water in First Nations Communities. This was a commitment made in the Strategy, before the release of the CESD report. In March 2006, the protocol was officially introduced as part of the Plan of Action for Drinking Water in First Nations Communities. The Plan also set up a Panel of Expert to develop options for an appropriate regulatory framework.

The Protocol

To assess whether the Protocol addresses the regulatory gap identified by the CESD, evaluators looked for established criteria. The Report of the Expert Panel on Safe Drinking Water for First Nations [NOTE 34] provides a list of all the elements covered by a comprehensive regulatory framework. These elements are: roles and responsibilities, non-piped water delivery systems, wells for individuals, water withdrawal and use, operator certification, monitoring, enforcement, appeal mechanism for regulatory decisions, reporting, design approvals, operating approvals for water and wastewater facilities procurement, construction and commissioning, emergency planning and response, drinking-water source protection, third-party audits and occupational health and safety. A comparison of the Protocol against this list shows that most elements are covered, but important elements related to coverage or enforcement are missing:

• The Protocol does not cover all the systems that the Report of the Expert Panel says a regulatory framework has to cover. It does not apply to wells for individual service or wastewater systems.
  
  
• The Protocol is implemented through water-funding agreements with First Nations. It thus falls to INAC to verify compliance. If there is non-compliance, retention of funding and, in theory, third-party management are potential recourses. INAC can also address underlying issues by funding system upgrades and enhancing operations and maintenance and training. As the Protocol is not a regulatory framework, there is no appeal mechanism. In fact, INAC is the de facto regulator; it defines the standards and, through third-party inspections (consultants hired by regional offices), verifies that the standards are met. However, INAC is also the main funding agency for the building and operation of water systems, and accountable for the overall results of the FNWMS. Therefore INAC lacks the independence and enforcement authority typically granted to regulators.
  
  
• The Protocol does not provide for permits and operation approvals. According to the Report of the Expert Panel: "all provincial jurisdictions in Canada require some form of permit or operation approval to run community water and wastewater systems". The only reference to an approval process is a note in the 2005 Risk Level Evaluation Guidelines for Water and Wastewater Treatment Systems in First Nations Communities [NOTE 35] regarding the applicable building and system-design standards that says: "actual provincial approvals/permits are not required, but our own standards require that all systems must meet provincial requirements, and this should in theory be capable of obtaining provincial approval/permits". The consequences of a system failing this theoretical test are not clear, although it is likely that it would receive a high-risk rating.

The Expert Panel

The second action, also part of the Plan of Action, was the appointment of an expert panel to provide options for a regulatory regime. The Report of the Expert Panel presented five options for creating a regulatory framework, only three of which were specified by the expert panel as viable:

• Parliament could enact a new statute setting out uniform federal standards and requirements;
  
  
• Parliament could enact a new statute referencing existing provincial regulatory regimes; or
  
  
• First Nations could develop a basis of customary law that would then be enshrined in a new federal statute.

Following the tabling of the expert panel's report, the Minister of INAC announced his intention to incorporate appropriate provincial laws into new federal legislation.

This approach was discussed before the Standing Senate Committee on Aboriginal Affairs. A number of positions were presented:

• Members of the expert panel and representatives of First Nations said that prior to imposing a regulatory framework, communities need to develop capacity required to abide by the regulations. Both said that sufficient resources must be provided and that sufficient capacity must be built before the implementation of a regulatory regime.
  
  
• INAC officials advised that a regulatory regime should be developed and implemented on a province-by-province basis. Existing provincial regulatory regimes would be modified, as required, to meet the needs of First Nations communities.

Notwithstanding this debate, it seems important that both the regulation authority and the operators (First Nations communities) are equipped to exercise their responsibilities. It is equally important that communities get the resources to develop their capacity and that the regulator acquires all the tools and attributes of a real regulation authority.

This section presents conclusions based on evaluation findings and recommendations for the renewal of the program.

Conclusions

The evaluation findings support the following conclusions about progress made, cost-effectiveness and relevance of the FNWMS.

Progress since 2003-2004

A review of performance monitoring systems and published reports shows that significant progress has been made since the Strategy's inception in 2003-2004.

Upgrades to the Systems

As of July 2007, there were 98 high-risk water systems, compared to 218 in 2003 (according to the National Assessment of Water and Wastewater Systems in First Nations Communities). [NOTE 36] The number of low-risk water systems increased from 185 (25 percent of total) to 288 (39 percent) and medium-risk systems increased from 337 (46 percent) to 355 (48 percent). The target identified in the Strategy's RMAF was to eliminate all high-risk systems by the third year. At the end of year four, the number of high-risk systems has been approximately cut in half.

However, some questions have been raised regarding these results.

• In 2005, the Commissioner on the Environment and Sustainable Development reported that the risk for health had not been reduced despite considerable investments in First Nations water and wastewater systems, including the ones supported by the Strategy.
  
  
• In 2007, First Nations representatives told the Standing Senate Committee on Aboriginal Affairs that a number of medium-risk systems were not far from becoming high-risk systems, because of the difficulty of training and retaining qualified operators.
  
  
• Performance-monitoring data suggests that most high-risk systems have improved to medium-risk rather than to low-risk.

It is not clear how this relates to the Strategy's objectives. The Strategy committed to ensuring that "water and wastewater facilities meet established federal standards and guidelines", although it measures performance in terms of risk levels. One can assume that a system rated as low-risk meets the appropriate guidelines and standards. It is less clear, however, whether medium-risk systems meet these guidelines. [NOTE 37] Moreover, standards and guidelines were better defined in 2006 with the introduction of the Protocol for Safe Drinking Water in First Nations Communities. An assessment of the extent to which the systems meet the standards and guidelines (as defined in the Protocol) will be possible only once all systems have undergone a complete inspection using the template provided in the Protocol.

Drinking-Water Monitoring

Delivery Capacity

Since 2002, under the FNWMS, capacity to deliver drinking-water monitoring in First Nations communities has increased. This increase is intended to promote more frequent monitoring, which is required to ensure safe drinking water, and it is evidenced by several factors. First, the number of EHOs has increased in all provinces and target ratios have been achieved in five of the seven regions. Most First Nations communities now have access to a qualified CBWM, and in five of the seven regions the percentage of communities with access to a monitor has increased. Second, the percentage of First Nations communities with assess to portable kits for onsite bacteriological analysis of drinking water rose. Third, a number of standards and protocols were produced and distributed to regional EHOs. These documents are intended to augment capacity to deliver water monitoring in First Nations communities.

Even with this program success, it should be noted that staffing EHO positions, especially in remote communities, is a challenge for HC. Considering existing EHO vacancies, anticipated retirements, and the high turnover rates among EHOs, effective recruitment and retention strategies are required to deliver core environmental-health programs in all First Nations communities. In addition, progress towards targets for CBWMs has been hindered by difficulties in recruiting and retaining qualified personnel for what is often a part-time position. And finally, with the distribution of standards and protocols to the regional EHOs, the level to which these documents are used after distribution is not known.

Frequency of Monitoring

Under the Strategy, drinking-water monitoring has improved. This improvement is documented by the steady increase in the total number of water samples for bacterial contaminants in First Nations communities since 2002. In 2006-2007, 700 percent more samples were taken than in 2002. This trend was primarily due to the increased availability of portable test kits and trained CBWMs. As a central part of its strategy, HC provided First Nations communities with onsite testing kits and the equipment needed to use them. It also provided funds to hire and train CBWMs.

In addition to the elevated number of samples taken, 44 percent of Community Water Systems were monitored weekly for bacterial parameters as required in the GCDWQ, and compliance with guidelines for chemical testing was high with 77 percent of Community Water Systems being tested annually. However, not all communities met the frequency standards for bacterial and chemical testing. Nor did they all employ the recommended quality-control procedure for onsite bacterial testing and send 10 percent of samples to external laboratories for verification. Additionally, compliance with frequency and quality control standards for bacterial and chemical testing varied among regions and water systems.

Identification of Drinking-Water Quality Problems

The number of identified and confirmed waterborne illnesses and drinking water advisories are poor indicators to use when assessing the intermediate outcome of monitoring (i.e., the ability to identify drinking-water quality problems and potential waterborne illness). Given that there were no confirmed cases of waterborne illness prior to the FNWMS, the subsequent absence of outbreaks cannot be attributed to the Strategy. Moreover, illness rates reflect the most negative outcome, and do not provide sufficient warning to take remedial action. The number of DWAs also fails as an indicator in that it lacks clear direction for improvement. The desired goal would be no DWAs–an objective shared by water treatment operators. From a public-health perspective, the goal of monitoring is a DWA whenever it is warranted.

Not only is the intermediate outcome not adequately measured by the current indicators, but it could also be argued that the outcome itself is not entirely appropriate for a monitoring strategy. A monitoring program should be assessed on how well it is designed and implemented. The ability to identify water quality problems stems from properly conducted and frequent sampling as well as accurate analysis. Results must also be communicated promptly and inform decisions about remedial actions. The evaluation revealed an increase in the number of samples taken. Additionally, the early warning databases now used in six of seven regions provide standardized reporting of results with electronic notification of problems. As such, these databases do not capture the recommendations made by the EHOs to the Chief and Council nor the amount of time take by the Chief and Council to act on the recommendation.

First Nations Community Confidence in Drinking Water

Public opinion research indicated that a substantial number of residents of First Nations communities lack confidence in their drinking water. Although the majority of residents believe drinking-water quality is good and their water is safe for consumption, a third still believes water quality is poor and unsafe to drink.

Operations and Maintenance

Evaluation findings suggest that operation and maintenance is one of the most important issues. As stated by a member of the Expert Panel to the Standing Senate Committee: "it is not the guidelines on water quality that ensure the safety of water; it is the quality of the operation and maintenance."

A review of the Capital Facilities Maintenance Program conducted for INAC concluded that investments in First Nations infrastructure were at risk of early rust-out due to lack of proper operations and maintenance. The CESD reported that INAC was not able to track all of the funds allocated to operation and maintenance. A review of a few annual inspection reports shows that in some cases Bands have not budgeted for the operation and maintenance of water systems and, when they do, budgets are often inadequate.

A review of the WATERS database shows that although some progress has been made towards the objective of "enhanced operations and maintenance practices", overall performance has been less than anticipated. Data for the last three years shows that the number of systems with maintenance management plans increased from 71, or 22 percent in 2005, to 128, or 35 percent, in 2007. The number of systems with emergency response plans increased from 129, or 15 percent in 2005, to 154, or 17 percent in 2007. The objective is that all systems have both management-maintenance and emergency-response plans.

A study commissioned by INAC clearly demonstrated that operations and maintenance costs for small and remote systems are higher than for systems serving larger populations. Costs are also influenced by a system's level of complexity or sophistication of the system; the more complex the system, the greater the cost of operation and maintenance. The study also established that the cost to operate and maintain an infrastructure project can account for 80 to 93 percent of lifecycle costs. Considering this, it appears that improvements in operations and maintenance practices would significantly increase the cost-effectiveness of the investments made in First Nations water systems.

Training and Certification

Closely related to operations and maintenance, is the issue of operator training and certification. The goal of the Strategy was to have each and every system operated by duly trained and certified operators. In July 2007, 38 percent of all system operators had at least Level I certification. The Standing Senate Committee Final Report on Safe Drinking Water for First Nations concluded that the efforts made by INAC to expand its training program were not sufficient and that a "comprehensive, long-term training program must be in put in place immediately". [NOTE 38]

According to the Expert Panel, the problem of insufficiently trained and certified operators is not specific to First Nations communities. In some provinces, (e.g., British Columbia, Manitoba and Ontario) it would appear that the proportion of small and rural communities served by fully accredited operators is about the same as it is in Aboriginal communities. There is a shortage of certified operators, and, given the time it takes to train and certify operators, this will not be solved in the short term. [NOTE 39] First Nations communities appear to have more difficulty retaining qualified and certified operators. The AFN told the Standing Senate Committee on Aboriginal Affairs that approximately 25 percent of plant operators in First Nations communities have left for municipalities or for private industry.

Another approach has been taken since the implementation of the 2006 Plan of Action: oversight by certified services providers. Oversight operators can assist when emergencies arise and help train onsite operators. A 2007 departmental progress report on the Plan of Action stated that oversight was being implemented in two phases: a 24-hour emergency line for technical support, operational since December 2006; and direct monitoring and review of local operators' logs and records. A highest level of oversight is being implemented through the Safe Water Operations Program, including full-time, onsite supervision by a certified operator. It is not clear however, if oversight is a viable long-term strategy or if efforts should be maintained in training local operators to get the appropriate certification level.

Comprehensive Standards

In the Strategy, INAC and HC committed to develop and implement a comprehensive set of clearly defined standards, protocols and policies, using the multi-barrier approach defined in From Source to Tap –the Multi-Barrier Approach to Safe Drinking Water. [NOTE 40]

HC has developed several standards and protocol documents including those that address topics such as monitoring water quality in distribution systems with five or more connections and cisterns, CBWM training, drinking water advisories, and reviewing water and wastewater project proposals from a public-health perspective. There is some indication that copies of these documents have been distributed to EHOs within the regions.

In 2006, INAC promulgated the Protocol for Safe Drinking Water in First Nations Communities to compile and clarify the standards for design, construction, operation, maintenance, and monitoring of drinking water systems. Since, April 2006, the Protocol has been included in the funding arrangements with First Nations for water systems.

A comparison of the protocol to the list of elements in a comprehensive regulatory framework [NOTE 41] showed that the majority of elements are included in the protocol. Missing elements relate to coverage and enforcement. For instance, the Protocol does not apply to private wells due to the fact that INAC does not fund the construction, operation and maintenance of private wells in First Nations communities. Moreover, given that the Protocol is implemented through funding agreements with First Nations, INAC must verify compliance via inspections by external assessors. In doing this, INAC assumes one of the critical functions of a regulatory authority.

Value for Money

Information was not available to measure the cost-effectiveness of capital expenditures in water systems (e.g., cost per service connection and per person served).

Available information suggests, however, that it costs more per connection to build, operate and maintain water systems in most of the communities targeted by the Strategy than in other communities in Canada. Factors identified as contributing to higher costs of the Strategy include:

• Higher costs to build, operate and maintain central-piped systems in small communities, First Nations or otherwise. Large economies of scale exist for water and wastewater systems; past a certain point, the cost per connection dramatically decreases as the size of the system increases. Such economies of scale are not available to small and remote communities.
  
  
• The design of the Strategy is likely to have had the unwanted effect of promoting non cost-effective solutions. Most small non-Aboriginal communities in Canada rely on private onsite services, such as private wells and septic tanks. These systems are not funded by the FNWMS, although they exist in many First Nations communities.
  
  
• Difficulties with operations and maintenance practices also decrease the cost-effectiveness of the Strategy. Operation and maintenance costs per connection are higher for small systems, and it is more difficult to hire and retain trained and certified operators in small and remote communities. The lack of proper operation and maintenance has been identified as a factor contributing to premature rust-out of infrastructure on-reserve.

The value for money assessment of the Strategy's monitoring component demonstrated that resources (i.e., CBWMs and testing kits) were secured at a cost lower than an alternative approach. Furthermore, cost-benefit estimates of water and sanitation in populations other than First Nations clearly illustrate the value of these types of environmental health programs, and indicate that averting disease outbreaks has substantial cost-benefit. However, evidence suggests that program expenditures are not adequately linked to program outputs and outcomes to fully measure value for money.

Relevance

The criterion used to assess relevance is whether a program provides an appropriate response to a continuing need. The Strategy was designed to address urgent drinking water and wastewater issues in First Nations communities. Obviously, not all the issues have been resolved. The question is how to determine if the overall approach remains valid and what, if any thing, needs to be changed or improved.

Based on the evaluation findings, evaluators concluded that:

• There is still a need to address urgent situations and to ensure that all systems meet the guidelines. It is not clear, however, if this need will be fulfilled when all high-risk systems have been eliminated, or when all systems are classified as low-risk. In any case, the approach implemented has demonstrated its capacity to have an impact on risk levels. This impact, however, might not be as great as suggested when considering only the decrease in the number of high-risk water systems.
  
  
• Although some progress has been achieved regarding the implementation of MMP and ERP, operations and maintenance remains one of the biggest risks to the safety of residents and the protection of water systems. The results observed with respect to operations and maintenance suggests that Strategy's objective will not be met if the current approach is maintained.
  
  
• Although INAC expanded the reach of its training program to include almost all First Nations (with the majority of the non-participating First Nations located in Northern Ontario where the Program was not available), the objective of having all systems operated by certified operators has not been reached. Results do not suggest that this target will be reached in the near future, except for Saskatchewan where an 88 percent rate of certification has been achieved and Manitoba, with 74 percent. The certification rates in all other regions fall below 50 percent. The report of the Standing Senate Committee on Aboriginal Affairs recommended that a new and more comprehensive training program be implemented. An internal assessment of the strengths and weaknesses of the existing Program suggests that it be allocated more resources.
  
  
• Although INAC developed and implemented the Protocol for Safe Drinking Water in First Nations Communities, providing a clear definition and presentation of the standards and guidelines applicable to First Nations water systems, some gaps remain. The Protocol does not cover wastewater systems, nor does it apply to private wells and septic tanks. There is no formal approval mechanism for water systems and wastewater system. It does not provide an appeal mechanism. As such, the Protocol lacks some of the necessary attributes of a comprehensive framework as defined by the Expert Panel.
  
  
• Since the Strategy was approved four years ago, HC's DWSP has focused on developing and enhancing delivery in the regions. The program remains highly relevant to the Government of Canada's priorities as there remains a compelling need to sample and test drinking water on-reserve. Without the Strategy, sampling and testing could return to its 2002 levels, which would be inadequate to protect human health.

Therefore, the evaluators conclude that there is a need to address pressing issues with drinking and wastewater systems in First Nations communities. The overall approach defined in the Strategy remains relevant although some elements need to be revised and improved.

Recommendations

Indian and Northern Affairs Canada

1. The Department should continue to provide assistance to First Nations for the building, operation, and maintenance of their water and wastewater systems as well as for the training of system operators. The Department should commit to address, in the short term, the remaining major risk issues with water systems.
   
   
2. Monitoring and reporting practices should be enhanced to ensure that the First Nations and INAC have reliable information about drinking-water and wastewater systems in First Nations communities. In particular, INAC should:
   
   
   1. ensure that all funded systems undergo complete annual on-site inspections according to the "Guide for Annual Inspections of First Nations Drinking Water Systems" in the Protocol for Safe Drinking Water in First Nations Communities;
      
      
   2. report more clearly on the extent to which systems meet established design, construction and water-quality standards; and
      
      
   3. collect data that support basic cost-effectiveness measurement, e.g., cost per connection and cost per person served for capital expenditures and for operations and maintenance.
      
      
3. INAC should
   
   
   1. revise its funding agreements with First Nations to ensure that funds awarded for operation and maintenance are used for that purpose; and
      
      
   2. take measures to ensure that the Maintenance Management Plans and Emergency Response Plans required under the Protocol are in place
      
      
4. INAC must significantly improve operator training and certification by substantially upgrading the Circuit Rider Training Program or by implementing a superior alternative.
   
   
5. The Department must address gaps in program design with a view to providing support for alternative cost-effective solutions such as wells and septic tanks.
   
   
6. Future policy development should consider the implementation of a regulatory framework that would separate INAC's roles as funding agency and de facto regulator. The option of having a separate federal department/agency or the provinces/territories or other entity (such as a First Nations organization/institution or aggregation), other than INAC, to enforce regulations for water and wastewater on-reserve should be explored.
   
   

Health Canada

Delivery Capacity

1. HC should develop recruitment and retention strategies to ensure that there is an adequate number of EHOs and CBWMs across the country to deliver core environmental health programming in all First Nations communities south of 60 degrees, since there are existing EHO vacancies, anticipated retirements, and the high turn over of EHO and CBWM positions.
   
   
2. Given the limited data collected on the distribution of standards and protocols to EHOs within the regions, HC should work with the REHMs to collect more targeted information related to the distribution and use of the standards and protocols produced by HC to assist with enhancing delivery capacity.
   
   

Frequency of Monitoring

• 3.HC should undertake a review to identify the barriers preventing all communities, in all regions, from complying with the required frequency and quality control standards as per the best management practices and the GCDWQ.
  
  
• 4.Quality control measures should be incorporated into a fully articulated quality assurance system. A strong QA system requires strong management support, clearly defined responsibilities and reporting requirements at each level of operations, detailed standard operating protocols, and a rigorous audit system to ensure compliance. Any QA system would require cooperation and support from staff at the national, regional and community levels to set out agreed upon roles and responsibilities.

Identification of Drinking-Water Quality Problems

• 5.HC should re-examine its Performance Measurement Strategy, associated outcomes, performance indicators and data sources. Specifically:
  
  
  1. HC should re-examine its outcomes, performance indicators and data sources. The Performance Measurement Strategy was based on information available at the onset of the FNWMS. Some outcomes were not sufficiently measurable and were not consistent with the theory underpinning the water monitoring program. Consequently, indicators were subsequently added to measure the success of the DWSP and support this evaluation. As well, whenever possible, performance indicators (e.g., sampling frequencies and QC compliance) should be directly gathered through centralized regional databases. A revised Performance Measurement Strategy would include appropriate and measurable outcomes and performance indicators as well as reliable and sufficient data sources to support program development and refinement.
     
     
  2. HC should improve its EHO Questionnaire to monitor progress in ensuring safe drinking water.
     
     
  3. HC should continue its work to identify and address waterborne threats to human health in First Nations communities. This work would allow HC to better identify and report on instances of disease outbreak as well as develop tools and materials for communities to reduce the health risk associated with waterborne threats.
     
     

First Nations Community Confidence in Drinking Water

• 6.HC should explore the issue of First Nations communities' confidence in their water.

Value for Money

• 7.HC should strengthen links between the Performance Measurement Strategy and program expenditures to support future value for money exercises and evaluation.

Appendix A: Performance Measures (RMAF)

Appendix B: List of Documents Cited in the Report

Assembly of First Nations, Expert Panel Report on Safe Drinking Water for First Nations. Position Paper.

Commissioner of the Environment and Sustainable Development (2005). Drinking Water in First Nations Communities, Report of the Commissioner of the Environment and Sustainable Development to the House of Commons, September 2005.

DPRA Canada (April 2006). Survey of First Nations Communities Regarding Drinking Water Advisories. Final Report.

Harrington, W., Krupnick, A.J., & Spofford, W. O. (1989). The economic losses of a waterborne disease outbreak. Journal of Urban Economics, 25, 116-137.

Hutton, G. & Haller, L. (2004). Evaluation of the Costs and Benefits of Water and Sanitation Improvements at the Global Level. Water Sanitation and Health, Protection of the Human Environment, WHO, Geneva.

Graham, J., & Fortier, E. (March 23, 2006). Building governance capacity: The Case of Potable Water in First Nations Communities. A paper presented at the Aboriginal Policy Research Conference, Ottawa.

Health Canada:

Health Canada and Indian and Northern Affairs Canada, First Nations Water Management Strategy Results-based Management and accountability Framework, July, 2003

Indian and Northern Affairs Canada:

Northwestern Health Unit (September 2006). Inspection Report on the Pikangikum Water and Sewage Systems.

Report of the Expert Panel on Safe Drinking Water for First Nations (October 2006). Volume 2 Legal Analysis, Indian and Northern Affairs Canada.

Risk Level Evaluation Guidelines for Water and Wastewater Treatment Systems in First Nations Communities, January 2005.

R.V Anderson Associates Limited (March 2007). Small Community Water/Wastewater Servicing Report.

Swain, H., Hrudey, S., & Loutitt, S. (October 2006). Report of the Expert Panel on Safe Drinking Water for First Nations. Volume 1, Indian and Northern Affairs Canada.

Swain, H., Hrudey, S., & Loutitt, S. (October 2006). Report of the Expert Panel on Safe Drinking Water for First Nations. Volume 2, Legal Analysis. Indian and Northern Affairs Canada.

Swain, H., Lazar, F., Pie, J., & Watertight (2005). The Case for Change In Ontario's Water And Wastewater Sector. Report of the Water Strategy Expert Panel. Ministry of Public Infrastructure Renewal.

The Honourable Dennis R. O'Connor (2002). Report of the Walkerton Inquiry. Volume 2. Published by Ontario Ministry of the Attorney General.

The Honourable Gerry St. Germain, P.C., Chair and The Honourable Nick Sibbeston, Deputy Chair, Safe Drinking Water for First Nations. Final Report of the Ottawa, May 2007, p. 3. Standing Senate Committee on Aboriginal Peoples.  

Value for Money Profile: Guidebook – Standard Version (Draft) (November 2006). Centre of Excellence for Evaluation, Results-Based Management Directorate, Expenditure Management Sector, TBS.

Appendix C: INAC Management Response and Action Plan

Original approved by: Claire Dansereau, December 18, 2007.

Senior Assistant Deputy Minister,

Socio-Economic Policy and Regional Operations

Appendix D: Health Canada's Management response and Action Plan

1. National Assessment of Water and Wastewater Systems in First Nations communities Summary Report, Ottawa, February 2003. The report is available on the INACwebsite. (return to source paragraph)
   
   
2. The March 22, 2007 Progress Report tabled in Parliament mentions 97 high-risk systems; one system was added to the list a few months later. (return to source paragraph)
   
   
3. The Honourable Dennis R. O'Connor, 2 Vol, Published by Ontario Ministry of the Attorney General, 2002. Report of the Walkerton Inquiry  (return to source paragraph)
   
   
4. National Assessment of Water and Wastewater Systems in First Nations communities Summary Report, Ottawa, February 2003. The report is available on the INAC website. (return to source paragraph)
   
   
5. From Source To Tap - The Multi-Barrier Approach To Safe Drinking Water (return to source paragraph)
   
   
6. Archives (return to source paragraph)
   
   
7. First Nations Water Management Strategy Results-based Management and Accountability Framework. Health Canada and Department of Indian Affairs and Northern Development, July, 2003. (return to source paragraph)
   
   
8. First Nations Water Management Strategy Results-Based Management and Accountability Framework, 2003, p. 15. (return to source paragraph)
   
   
9. Commissioner of the Environment and Sustainable Development, 2005 Report of the House of Commons,Chapter 5: Drinking Water in First Nations Communities, Minister of Public Works and Government Services Canada, 2005,41 p.(return to source paragraph)
   
   
10. Draft terms of references for a summative evaluation of the Strategy had been developed under the supervision of the program managers in INAC and HC. In October 2006, Treasury Board Secretariat requested that this evaluation be conducted under the supervision of the Evaluation Services Directorate. The final terms of reference for this evaluation were presented to Treasury Board Secretariat officials. (return to source paragraph)
    
    
11. The list of performance indicators from the RMAF is provided as Appendix A. (return to source paragraph)
    
    
12. A new database is being implemented by INAC to manage its infrastructure programs: the Integrated Capital Management System (ICMS). ICMS should resolve most of the issues experienced in this evaluation; data from the Annual Inspection reports will be included in that database. (return to source paragraph)
    
    
13. A third performance indicator is the number of water and wastewater systems upgraded, but is it too similar to the one measuring the immediate outcome–the number of upgrades made to systems. If one must distinguish between the two performance indicators, the latter should be the output indicator, and the former the outcome indicator. (return to source paragraph)
    
    
14. Anticipating on results presented further in the report, it could be argued that the difference between the number of plans developed, approved and implemented might be a measurement useful to identify the source of the problem with MMP and ERP. (return to source paragraph)
    
    
15. Although it is monitored by regional offices, it was not possible to get a report from the WATERS database listing all the facilities with a trained operator. (return to source paragraph)
    
    
16. Link not available. (return to source paragraph)
    
    
17. The link is not available  (return to source paragraph)
    
    
18. Although the WATERS database tracks the works to be done and their completion, the Program could not generate a report with a count of all the capital projects for the systems during the period. The use of the proxy was validated by program officials. (return to source paragraph)
    
    
19. At the time of writing this report, INAC officials were researching liability-insurance options in hopes of securing a group rate for circuit riders. (return to source paragraph)
    
    
20. This number differs from the December 2006 Progress Report. In the present evaluation, non-active operators and double entries have been eliminated. (return to source paragraph)
    
    
21. The Honourable Gerry St. Germain, P.C., Chair and The Honourable Nick Sibbeston, Deputy Chair, Safe Drinking Water for First Nations. Final Report of the Standing Senate Committee on Aboriginal Peoples, Ottawa, May 2007, p. 3. (return to source paragraph)
    
    
22. Harry Swain, Proceedings of the Standing Senate Committee on Aboriginal Peoples Issue 16 - Minutes of Proceedings, Ottawa, Tuesday, May 15, 2007, (34). Proceedings of the Standing Senate Committee on Aboriginal Peoples  (return to source paragraph)
    
    
23. The AFN is currently doing a study on operator retention. With financial contributions from INAC, it is currently conducting a study on the retention of water operators in First Nations communities. (return to source paragraph)
    
    
24. The March 22, 2007 Progress Report tabled in Parliament mention 97 high-risk systems. Between that date and July 2007, the rating of one system was elevated. (return to source paragraph)
    
    
25. The difference between that number and our subset number of high-risk systems for 2005-2006 is explained by the fact that regions had until March 31, 2006 to enter their results, resulting in the need for changes to some numbers to complete the reporting. (return to source paragraph)
    
    
26. Office of the Auditor General of Canada (return to source paragraph)
    
    
27. A costly response to a health and safety issue 18,8 (return to source paragraph)
    
    
28. R. V. Anderson Associates Limited, Small Community Water / Wastewater Servicing Report, Ottawa, Final Report, March 2007. (return to source paragraph)
    
    
29. Ibidem. (return to source paragraph)
    
    
30. Ibidem. (return to source paragraph)
    
    
31. Value for Money Profile: Guidebook – Standard Version (Draft). Centre of Excellence for Evaluation, Results-Based Management Directorate, Expenditure Management Sector, TBS, November 2006 (return to source paragraph)
    
    
32. "Evaluation of the Costs and Benefits of Water and Sanitation Improvements at the Global Level" (2004) Hutton, Guy and Laurence Haller, Water Sanitation and Health, Protection of the Human Environment, WHO, Geneva (return to source paragraph)
    
    
33. "The Economic Losses of a Waterborne Disease Outbreak" Winston Harrignton, Krupnick, Alan J, Spofford, Walter O Jr. Journal of Urban Economics 25, 116-137 (1989). (return to source paragraph)
    
    
34. H. Swain, Louttit S., Hrudey S. Report of the Expert Panel on Safe Drinking Water for First Nations. Vol 1, November 2006, vol.1, pp. 35-39. (return to source paragraph)
    
    
35. Risk Level Evaluation Guidelines for Water and Wastewater Treatment Systems in First Nations Communities, 2005, p. 10. (return to source paragraph)
    
    
36. National Assessment of Water and Wastewater Systems In First Nations Communities - Summary Report (return to source paragraph)
    
    
37. And according to some program officials, it is possible for a high-risk system to meet the existing guidelines and standards, although this is surely not a widespread situation. (return to source paragraph)
    
    
38. Safe Drinking Water for First Nations. Final Report of the Standing Senate Committee on Aboriginal Peoples, Ottawa, 2007, p. 3. The Committee refers to the Circuit Rider Training Program, where a qualified trainer provides onsite one-on-one training to many communities. (return to source paragraph)
    
    
39. Harry Swain, Proceedings of the Standing Senate Committee on Aboriginal Peoples Issue 16 – Minutes of Proceedings, Ottawa, Tuesday, May 15, 2007, (34). (return to source paragraph)
    
    
40. From Source to Tap. (return to source paragraph)
    
    
41. Report of the Expert Panel on Safe Drinking Water for First Nations, vol. 1 p. 35-39. (return to source paragraph)