Archived - Evaluation of Grants for the Promotion of Fire Protection Awareness in Band and Federally Operated Schools

Date: February 24, 2010
(Project Number: 08045)

PDF Version (404 Kb, 18 Pages)

Table of Contents

• List of Acronyms
• Executive Summary
• Management Response / Action Plan
• 1. Introduction
• 2. Evaluation Methodology
• 3. Evaluation Findings - Relevance
• 4. Evaluation Findings - Performance
• 5. Evaluation Findings - Program Design and Delivery
• 6. Conclusions and Recommendations
• Appendix A - 2008-2009 Fire Prevention Winning Posters

Executive Summary

The following report presents the findings of an Evaluation of Grants for the Promotion of Fire Protection Awareness in Band and Federally Operated Schools.

The evaluation was conducted by the Evaluation and Performance Measurement Review Branch of the Indian and Northern Affairs Canada (INAC) and will inform the program's terms and conditions, which expire on March 31, 2010.

Background and Description

The purpose of the grant program is to promote fire protection awareness at the community level by way of a Poster Contest aimed at children and youth.

The grant program is currently managed within the Community Infrastructure Branch under the Strategic Initiatives Directorate. The Poster Contest is administered, on INAC's behalf, by the Aboriginal Firefighters Association of Canada with which INAC has a service agreement.

The $136K grant program has its own set of terms and conditions and authority.

Evaluation Methodology

The evaluation was conducted between January and February of 2010. Findings are based on:

• a review of program documentation;
• a review of literature on fire prevention related programming (including international literature); and
• interviews (total n=6) with key informants, including INAC program management (n=2) and regional organization stakeholders (n=4).

The main areas of findings for this evaluation are related to relevance and performance. Program design was also reviewed.

Relevance

The document review and literature demonstrate that there is a need to promote fire protection awareness to First Nations communities and that the grant's objective of promoting fire protection awareness in First Nation'scommunities is in keeping with the priorities of the Department, the Government of Canada and is also relevant to broader fire prevention initiatives, both nationally and internationally.

Effectiveness

Interview evidence suggests that more can be done to raise awareness beyond the school level and into the community at large. Evidence from key informant interviews suggest that participation rates amongst the youth cohort is not as high as participation rates amongst the younger age cohorts.

Document review and interviews suggest that the initiative has had some success in achieving its intended outcomes; however, the precise impact of the initiative is difficult to determine since it is unknown whether associations are consistently tracking participation levels by schools and by age cohort as there are currently no requirements for data collection collected that can measure participation levels.

Program Design

Overall, interviews with key informants indicated that the processes involved in the delivery of the program are working well.

Although the program is closely aligned with education programming, a review of the Department's Program Activity Architecture and other documents, indicate that the program could eventually be streamlined and situated under the authority of the Capital Facilities Maintenance (CFM) Program.

Recommendations

Based on the findings of the evaluation, the following recommendations are made:

• Identify indicators for tracking performance information on participation levels (by school and age category) to be collected by organizations involved in the delivery of the grant program and review the program's target audience and reach in light of the absence of performance information on participation rates.
• Work toward streamlining the program's authorities and situating them under the CFM Program.

Management Response / Action Plan

1. Introduction

1.1 Overview

The purpose of this evaluation is to conduct a review of Indian and Northern Affairs Canada (INAC)'s Fire Prevention Poster Contest, which is a grant program aimed at promoting fire protection awareness in band and federally operated schools.

This review is required for the renewal of the program's terms and conditions, which will expire on March 31, 2010.

1.2 Program Profile

1.2.1 Background and Description

Since 1990, INAC has been responsible for funding the Fire Prevention Poster Contest through a grant program aimed at promoting fire protection awareness in band and federally operated schools.

The program encourages children and youth to be involved with fire protection and prevention by participating in developing a poster, which includes the possibility of winning a monetary reward as an incentive.

The program has a wide reach, which potentially includes thousands of First Nation youth across Canada each year, in eight regions. Eligible recipients are restricted to First Nation students attending band or federally operated schools.

Due to the regional nature of the poster contest, regional fire fighting associations play an important role in helping to deliver the program. While, the program is still managed by INAC, the Department has a service agreement with the Aboriginal Firefighters Association of Canada (AFAC), a national organization, to deliver the program.

The fire poster competition includes the following three categories of contestants, with judging and prizes awarded in each category:

• Kindergarten to grade 2;
• Grades 3-6; and
• Grades 7-12.

The Poster Contest process is initiated by AFAC sending out call letters requesting sponsoring regional fire prevention agencies to contact the schools with the rules and guidelines for the Poster Contest.

Category winners advance progressively through three different levels of the competition. The first level involves competition at individual schools. One winner is selected, by the school principal, from each age category at each participating school. Participation in the competition is voluntary and at the discretion of the school principals.

The second level of competition is at the regional level where officials from the regional association direct the competition and judging will result in one winner per category per region being selected.

The third and final level is held at the annual AFAC meeting. Each winner competes in one of the three age categories. A maximum total of eight national prizes (first, second, third and five honourable mentions) for each age category ensures winners from each participating region win a prize, thereby maximizing the fire protection promotional value of the competition.

The program has a total annual cost of $142K. The main program costs of $136K are the grants for monetary prizes and the reimbursement of travel expenses for winning participants, which are made directly to the winning participants. [Note 1] There are additional operating costs to cover expenditures incurred by persons participating in the judging process ($6,000) and other operating expenditures ($2,000).

Over the five-year period under review (April 2005 – March 2010), program resources have totalled $712K.

The maximum prize amount a student may receive is a combination of $500 for winning at both the school and regional competitions. For the national awards, costs for first prize winners in each category and their chaperons are limited to $4,333 (includes travel, accommodation and meal allowances) whereas the maximum total for the regional category is $1450.

Each region is also limited in that regional costs for travel, accommodation and meal allowances are limited to $3,000 per region and the prizes costs are limited to $12,000 per region.

The program is currently aligned with INAC's Education and Social Development Programs and Partnerships Sector, under the Community Infrastructure Branch. It is currently managed under the Strategic Initiatives Directorate.

2. Evaluation Methodology

2.1 Evaluation Scope and Timing

No evaluation or audit was scheduled for this program at the time the program's authorities were renewed in 2005. This is most likely due to the grant-based nature of the program (in that grants do not require reporting) and since the program can be considered to be relatively low-risk (i.e.. level of funding and the level of public sensitivity to this program is relatively low). However, the Evaluation, Performance Measurement and Review Branch (EPMRB) decided that it would be prudent to evaluate the program prior to the program's authorities expiring on March 31, 2010.

2.2 Evaluation Issues and Questions

The main lines of enquiry and areas of findings for this evaluation are related to the relevance and effectiveness of the Poster Contest and, to a lesser extent, program design and delivery:

Relevance

• Is there a need for INAC to promote fire protection awareness to First Nation's communities? Is the aim of the grant consistent with the priorities of the Department and the Government of Canada? To what extent does it complement other initiatives?

Performance

• To what extent is the Poster Contest contributing to raising awareness at the community level? How can the Poster Contest further increase awareness of fire protection? Does the Poster Contest reinforce other fire awareness initiatives administered by First Nations (i.e. "Learn not to Burn" and the Wisdom of the Fire")?

Design and Delivery

• Is the authority appropriately situated? Are there any issues with regard to how the program is currently delivered

2.3 Evaluation Methodology

The evaluation was completed in-house by the EPMRB between January and February of 2010. Methodology included:

• a review of program documentation;
• a review of literature on fire prevention related programming (including international literature); and
• interviews (total n=6) with key informants, including INAC program management (n=2) and regional organization stakeholders (n=4).

Performance data on participation (by school and age category) in the Poster Contest was not available for the purposes of the evaluation and is a limitation of the study.

3. Evaluation Findings - Relevance

3.1 Need Amongst First Nations Communities

A review of document and literature indicate that there is a need to promote fire protection awareness to First Nation's communities.

Fire losses in the form of deaths, personal injuries and destruction of property in First Nation communities, particularly those on remote First Nations lands, far exceed those in comparable off-reserve communities. [Note 2]

The 2002 fire death rate in First Nation communities, expressed as the number of deaths per 100,000 population was 3.2. A recent report by the Canada Mortgage and Housing Corporation found that the death rate from fires in Aboriginal communities is more than 10 times higher than in the general population and that the fire rate is more than double (Canadian Press, January 15, 2009)

The incidence of fire losses in terms of fire deaths and property loss on First Nations is also high and substantially above the Canadian average. The most effective way to lower the number of deaths caused by fires is to prevent fires from taking place in the home. In addition to saving lives, the prevention of fires reduces property losses and reduces insurance costs.

3.2 Consistency with INAC and Government of Canada priorities

Documents were reviewed in order to assess whether the aim of the grant is consistent with the priorities of the Department and the Government of Canada.

Review of documentation and interviews with program management indicate that the program contributes to the achievement of INAC objectives of aiming to achieve individual and family well-being (under "the People") and in aiming to achieve economic well-being and prosperity (under "the Economy"). Given that the INAC is the primary department responsible for First Nations on reserve, it is fitting that a grant program for raising awareness in fire protection amongst First Nations falls under INAC authority.

The program is further aligned with broader government priorities related fire prevention in the context of health and safety. This is exemplified through Human Resources and Skills Development Canada (HRSDC)'s Fire Protection Services, which is the federal government's technical authority in the area of fire protection, providing services related to the administration of major public buildings within Canada and including First Nation Communities.

In addition to providing technical support, Fire Protection Services also offers two types of grants for the purpose of fire prevention activities. One grant is to assist Fire Prevention Canada (FIPRECAN) [Note 3] in its fire prevention education objectives of keeping the Canadian public (including First Nations communities) informed and aware of the causes of fire and fire-related preventive measures. The funding for the grant, however, is minimal ($7,000 per fiscal year). [Note 4] The other grant offered through HRSDC is for fire safety organizations aimed at supporting the promotion of national public awareness and research. Funding may not exceed $19,000 in any fiscal year, to be paid to one organization or in smaller amounts to several participating organizations.

Nunavut has also actively engaged in fire prevention activities, normally around the week of National Fire Prevention Week in October. The Nunavut Office of the Fire Marshal, for example, its own version of a colouring design and essay contests for students. [Note 5]

3.3 Complementing Other Initiatives

Seeking to address fire protection issues more effectively, officials from INAC and from the regional member organization of AFAC held a one-day strategic planning workshop (January 2009) to develop a draft strategic plan for improving fire prevention, awareness and protection in First Nation Communities

Key strategic objectives identified for the plan for the short term include increasing awareness of fire prevention and improving data integrity for more accurate reporting. [Note 6] Short-term activities identified in the development of strategies and solutions include improved fire education (i.e. developing education kits) and raising awareness in community and in schools through AFAC existing material (Wisdom of the Fire and Learn to Burn) and focusing on children and youth to develop a prevention program, recruit fire fighters by developing incentives for volunteer fire fighters, improve data collection and integrity (re: participation levels, which can link to reporting on fire loss reports).

The fire protection strategy would be only one component of a comprehensive set of activities for fire protection. One other major undertaking would be to develop an effective community of practice, improved communication and clear roles for all stakeholders, including fire associations at the regional or provincial level and other federal departments. Other activities would be to implement an awareness campaign aimed at Chief and council, providing school kits for elementary and secondary schools, and continuing the national fire poster contest and fire fighter competition. [Note 7]

Finally, while the grant program is a far-cry from a full-fledged social marketing campaign, it is relevant in the context of other fire prevention initiatives, which are being employed internationally. Vision 20/20 is a United States national strategic planning process to look at the nation's fire prevention efforts. Major strategies with action items include increasing advocacy for fire prevention and conducting a national fire safety education/social marketing campaign, which includes increasing educational outreach through school/youth based programs. [Note 8]

Therefore, based on evidence reviewed from the available documents and literature, there is a need to promote fire protection awareness to First Nations communities. The grant's objective of promoting fire protection awareness in First Nation's communities is in keeping with the Department's priorities and the Government of Canada's priorities. It is also relevant to broader fire prevention initiatives, both nationally and internationally.

4. Evaluation Findings - Performance

The evaluation addressed the questions of whether the Poster Contest is contributing to raising awareness at the community level and how it can further increase awareness of fire protection.

4.1 Contribution to raising awareness at the community level

The goal is to improve children's and youth awareness concerning fire protection and prevention and support INAC's responsibility in raising the level of fire protection awareness and knowledge at the community level.

Document review and interviews suggest that the initiative has had some success. For example, the Ontario First Nations Technical Services Corporation (OFNTSC) cites the fire safety prevention poster contest as becoming "an integral part of the OFNTSC Fire Safety Program", serving as a vital educational tool for fire protection and prevention awareness.  OFNTSC further claims that "the program delivers an important message to students and families alike."

In achieving its intended outcomes, however, the precise impact of the initiative is difficult to determine since INAC currently has no requirements for data collection on participation, which serves as a limitation to assessing performance. Although interviews with two organizations indicate that this information may be available, the collection of participation by school and by age category is not consistently collected.

There is some evidence, both from document review and interviews, which indicates that participation levels are not meeting expected levels. According to evidence from the document review, at the 2009 Poster Contest, first place winners were not in attendance at the annual event, therefore, there would be a surplus of funds in the travel component of that portion of the grant. The unused funds were redirected into other relevant fire prevention related programming, such as the National First Nations Fire Fighters Competition.

Although the objectives for raising fire prevention awareness of the overall program were still being met, it speaks to the fact that participation levels in the Poster Contest were not being maximized. INAC has recognized this and has indicated that it plans to work closely with AFAC and establish a protocol for the Poster Contest to ensure that award winners are present at future awards ceremonies and the program reaches the maximum number of children possible. [Note 10] However, without accurate performance information as it relates to participation levels by individual school and age cohort, it is difficult to assess the extent to which participation is not being maximized.

Evidence from the key informant interviews indicate that the Poster Contest results in increased awareness at the community level. This is exemplified through the fact that the winning posters are displayed around various community buildings, and relying on the assumption that the children will communicate their understanding more broadly at the community level. However, literature review indicates that another means of raising awareness would be through a national, social marketing campaign or broader strategy on fire prevention.

Evidence from at least two key informants suggested that participation amongst the older youth (grades 7-12 cohort) is not as high as participation rates amongst the younger age cohorts. Suggestions for improving the program were to have something more challenging than the poster contest geared for the older age cohort (to increase participation levels).

Therefore, while there is some evidence that indicates the poster contest is contributing to raising awareness at the community level, there is some evidence to suggest that participation in the Poster Contest is not maximizing funds identified for the purpose of the Poster Contest. However, funds are still being used for the overall objective of raising awareness of fire prevention through other fire prevention activities.

4.2 Reinforcement of other initiatives

The evaluation addressed the questions of whether the Poster Contest reinforces other fire awareness initiatives administered by First Nations (ie. "Learn not to Burn" and the Wisdom of the Fire") and to what extent is does it complement other initiatives?

The contest is also supposed to provide an opportunity to reinforce other fire awareness initiatives, administered by First Nations and not directly funded by INAC, such as school visits by the fire protection personnel, "Learn Not To Burn" – a National Fire Protection Association Program used since the mid-eighties and "Wisdom of the Fire" - curriculum developed in part by AFAC, Assembly of First Nations and other federal departments.

The Poster Contest is considered by most interviewees to be successful in reinforcing these initiatives, as it is often used to complement other programs as part of the school curriculum, normally around Fire Prevention Week. However, one interviewee was not sure whether the other initiatives were being implemented in schools as there is no information being collected to indicate this.

5. Evaluation Findings - Program Design and Delivery

5.1 Situation of Authority

The grant program is currently managed within the Education and Social Development Programs and Partnerships Sector under the Community Infrastructure Branch (CIB), Strategic Initiatives Directorate.

One of the key upcoming activities of the Directorate is to "create a fire prevention, awareness and protection strategy for First Nation communities. The broad objective of the strategy would be to affect a decrease in fire-related losses (deaths, injuries and facility damage) by focusing on fire awareness and prevention in the short term. Another goal would be to gain improved data on fire associations and tribal councils. [Note 11]

While the CIB is responsible for development of policies and directives for program delivery in the area of community infrastructure, the Operations and Planning Support Branch is responsible for implementing the policies, procedures and directives by the CIB, as well as developing reporting requirements and managing program data and performance measurement.

Although the program is closely aligned with education programming, it is not delivered by the school system. A review of the Department's Program Activity Architecture and other documents, indicate that the program could eventually be streamlined and situated under the authority of the Capital Facilities and Maintenance (CFM). The CFM Program provides funding for housing and community infrastructure, including capacity-building for, amongst other things, fire protection awareness and is delivered in part by the CIB and the Operations and Planning Support Branch.  The CFM Program also contributes to the Government of Canada outcome area of healthy Canadians.

5.2 Program Delivery

With regard to how the program is being delivered, the Poster Contest is administered through AFAC with which INAC has an agreement (April 2009) for the purpose of delivering and administering the AFAC fire and emergency prevention program. AFAC is a non-profit organization governed by a volunteer Board of Directors with regional representation.

AFAC receives funding to provide the administrative support to the National Fire Safety Poster Contest, as well as other areas related to fire prevention and awareness, such as the National Aboriginal Firefighters Competition, and research and dissemination of other information related to fire safety prevention programming.

AFAC objectives associated with the Poster Competition include:

• Administering the program delivery and funds provided for awards and support costs;
• Establishing direction, location, activities, for the Poster Competition (and other national fire/emergency related competitions);
• Providing management, project coordination and administrative support in carrying out the Poster Competition;
• Monitoring and providing recommendations on the progress of the poster competition and providing reports to funders; and
• Assisting with presentations in schools and learning centres as required.

Overall, interviews with key informants indicated that the processes involved in the delivery of the program are working well.

6. Conclusions and Recommendations

6.1 Recommendations

Based on the findings of the evaluation, the following recommendations are made:

• Identify indicators for tracking performance information on participation levels (by school and age category) to be collected by organizations involved in the delivery of the grant program and review the program's target audience and reach in light of the absence of performance information on participation rates.
• Work toward streamlining the program's authorities and situating them under the CFM Program

Appendix A - 2008-2009 Fire Prevention Winning Posters

Footnotes:

1. Eligibility is restricted to First Nation students attending band or federally operated schools and their chaperons (for travel). Terms and Conditions. Page 12-13. (return to source paragraph)
2. Report on INAC-AFAC Joint Strategic Planning Workshop. January 22, 2009 (return to source paragraph)
3. FIPRECAN is a national non-profit organization which aims to increase visibility and awareness of fire prevention nationally, through educating the public, including organizing the yearly national launch of Fire Prevention Week. (return to source paragraph)
4. HRSDC. Grant to Fire Prevention Canada (return to source paragraph)
5. Hansard Official Report. Nunavut Canada. Legislative Assembly, 6th Sessions. 2002. (return to source paragraph)
6. Report on INAC-AFAC Joint Strategic Planning Workshop. Institute on Governance. January 22, 2009. (return to source paragraph)
7. Ibid. (return to source paragraph)
8. Vision 20/20 National Strategies for Fire Loss Prevention, October 6, 2008. page 6 (return to source paragraph)
9. page 2. Summary Report, Aboriginal Firefighters Association of Canada Conference and Competition. (return to source paragraph)
10. Report on INAC-AFAC Joint Strategic Planning Workshop. Institute on Governance. January 22, 2009. page 3. (return to source paragraph)