Archived - Evaluation of the Advocacy and Public Information Program (APIP)

Date: February 13, 2009

PDF Version   (134 Kb, 42 pages)

Table of Contents

• Initialisms and Abbreviations
• Executive Summary
• 1.0 Introduction
• 2.0 Program Profile
• 3.0 Evaluation Methodology
• 4.0 Findings: Relevance
• 5.0 Findings: Design and Delivery
• 6.0 Findings: Success and Impacts
• 7.0 Findings: Effectiveness
• 8.0 Conclusions and Recommendations
• Appendix A - Comparative Analysis of Estimate Residential School Attendees and Common Experience Payment
• Appendix B - Management Response / Action Plan

Initialisms and Abbreviations

Executive Summary

This report presents the findings of the evaluation of the Advocacy and Public Information program, undertaken by the Evaluation, Performance Measurement and Review Branch (EPMRB) of Indian and Northern Affairs Canada (INAC). The evaluation sought to assess the Program's relevance, program design and delivery, effectiveness and success and impact on end users.

Context

The Indian Residential School Settlement Agreement came into effect on September 19, 2007. The Advocacy and Public Information program was established to inform former students and the general Canadian population about the Settlement Agreement and the impact of the Indian Residential Schools legacy on Aboriginal communities. These activities were to be undertaken as part of a larger communications strategy to disseminate information and educate these target audiences.

There are six key components of the Settlement Agreement that have related programs that the Advocacy and Public Information program disseminates information to former students about benefits available. These include:

• Common Experience Payment
• Independent Assessment Process
• Truth and Reconciliation Commission
• Commemoration Program
• Healing Fund (via the Aboriginal Healing Foundation)
• Availability of Mental Health and Emotional Support Services [Note 1].

Each of these Programs, along with the Courts and Indian and Northern Affairs Canada's (formerly Indian Residential Schools Resolution Canada) Communications Branch plays a complementary role in disseminating information. The Program Interlocutors are responsible for communications on their area of responsibility, while the others share information on the overall Settlement Agreement and all related programs. The Advocacy and Public Information program was the mechanism by which grassroots organizations were engaged to support outreach activities.

Program Objectives and Expected Results

The primary objectives of APIP are to encourage information sharing on the Settlement Agreement and the benefits for former students, their families and their communities; to support opportunities to enhance service delivery; and to help sensitize Canadians to the impact of the legacy of Indian Residential Schools on Aboriginal Communities.

The expected result was to help ensure former students of Indian Residential Schools have complete access and equal participation in the Settlement Agreement. As per the overall ambition of the Settlement Agreement, to bring about a lasting resolution to the legacy of Indian Residential Schools, it was anticipated that such a result would in turn contribute towards reconciliation between Residential School survivors and the Government of Canada.

Program Activities and Contribution Recipients

APIP was allocated $10 million, under a 2-year allotment ending March 31, 2009. The program is seeking an extension for a further four years. The 2007-08 fiscal year funding allocation, of $6M, was fully invested to support 27 initiatives, while the 2008-09 allocation of $4M has been committed to 15 projects, with an additional 7 projects still in negotiation. Between April 2007 and November 2008, 42 contribution agreements were signed with 25 organizations, predominantly Aboriginal. Contributions ranged from $25,000 to $603,240 with the average being $200,866.

Contribution agreements reflect a balance of demographic and geographic representation of intended audiences. These included First Nations, Inuit, and Métis; on- and off-reserve, rural and urban communities; and the general Canadian population.

Projects in each fiscal year, focused on disseminating information pertaining to programs related to the Settlement Agreement that were being rolled out within that respective year. In fiscal year 2007-08, projects focused on disseminating information on the Common Experience Payment, while in 2008-09 projects were to focus on the Truth and Reconciliation Commission, the Independent Assessment Process and Commemoration.

Evaluation Methodology and Limitations

The data collection strategy was comprised of multiple lines of evidence that were triangulated where possible. Research included document and literature review; a review of administrative data and information; review of secondary data sources; key informant interviews with contribution recipients, departmental officials and former students; an electronic survey of contribution recipients; and four case studies.

Initially, a survey of former students was planned. Upon further consideration of the sensitivity of the subject matter and the availability of secondary data sources, it was determined that the use of existing data would support evaluation interests.

The lack of a documented performance measurement strategy including baseline data, a program specific logic model, targets, performance indicators and results focused reporting limited the capacity of the evaluation to draw substantive conclusions on program success and cost-effectiveness.

Additionally, scheduling challenges limited the scope of the evaluation's fieldwork. Given the diversity of parties engaged in disseminating information on the Settlement Agreement, the evaluation could not draw conclusive findings regarding attribution of results achieved to the Advocacy and Public Information program. Lack of performance information also contributed to this challenge.

Conclusions

The implementation of the Indian Residential Schools Settlement Agreement remains a federal and departmental priority. With current delays in the implementation of the Settlement Agreement, there continues to be a need for communication activities directed to survivors, their families and communities and the broader Canadian population.

APIP's strategy of engaging credible and trusted Aboriginal organizations was found to be appropriate given the sensitivity of the information to be shared. It is also an appropriate response to the urgency of disseminating information about the Settlement Agreement. One challenge, however, is that no contingency plan is in place to inform individuals and communities when capable and willing organizations at the community level cannot be identified in a timely manner.

The lack of clarity in the program's expected results and performance measurement gaps severely inhibit the Program's capacity to monitor performance, measure the achievement of expected results, or identify issues or factors which may differentially affect access to information to and participation in benefits from the Settlement Agreement.

In addition to performance measurement gaps, the short time frame in which APIP has been operating also limited the potential for assessing impacts at this time. A number of promising practices were identified in terms of maximizing the reach of communication activities. Moreover, according to an analysis of departmental data, APIP project funding seems to be distributed in proportion to regional needs, but no strong correlation was found between funding and uptake. The evidence suggests that future initiatives should take additional steps to ensure clarity in information disseminated and ensure mechanisms are in place to identify communication related issues affecting applications.

Efforts are being taken to reduce overlaps and gaps and to enhance coordination in order to enhance the results of Residential School related communications. With more Settlement Components coming on-line, and some potential opportunities for overlap being noted, now is the time to ensure the roles and responsibilities are clear and complementary, and structured or streamlined in such a manner so as to best facilitate uptake.

Recommendations

The evaluation findings support the following recommendations:

1. In light of the revised implementation schedule for various Settlement Components, INAC should consider continuing support for communication activities, based in part on the APIP model, to assist in informing former students and Canadians about the Settlement Agreement;

2. Clarify roles and responsibilities of the various parties involved in disseminating information about the Settlement Agreement so as to ensure cost-effectiveness in the delivery of information and reduce the potential for duplication and overlaps; and

3. Depending on the approach adopted, develop a results based management tools and performance measurement strategy, that is mindful of reporting burden, yet includes:

• a program specific logic model demonstrating links to the Settlement Agreement objectives and INAC's program activity architecture;
• a needs assessment exercise which draws upon existing data (e.g., baseline study, program uptake rates) disaggregated by location, age and gender;
• performance measurement indicators and targets;
• formalized recipient selection and assessment criteria which are clearly aligned with program objectives (particularly with respect to reach)
• results-based reporting requirements; and
• a client survey to be applied for all activities undertaken (i.e. clarity and utility of information provided, change in understanding, and ability to act based on information provided).

1.0 Introduction

1.1 Introduction

This report presents the findings for the evaluation of the Advocacy and Public Information program (APIP), undertaken by the Evaluation, Performance Measurement and Review Branch (EPMRB) of Indian and Northern Affairs Canada (INAC). The evaluation focuses on the Program's relevance, success and effectiveness and is intended to inform future programming.

The Advocacy and Public Information program, established in 2007, was designed to support the implementation and results of the Indian Residential Schools Settlement Agreement by informing former students and Canadians about the Agreement and the impacts of the Indian Residential Schools legacy.

1.2 Context: The Settlement Agreement [Note 2]

Indian residential schools began operating in Canada around 1800 and were largely operated by Anglican, Roman Catholic, Methodist and Presbyterian churches. As early as 1874, the federal government was involved in their development and administration in line with requirements of the Indian Act. Over time, 130 schools were distributed across the country, with the exception of Newfoundland, New Brunswick and Prince Edward Island. On April 1, 1969, the Government assumed full responsibility for the schools. While most had ceased to operate by mid-1970, the last school closed in Saskatchewan in 1996.

Throughout 1996, the Royal Commission on Aboriginal Peoples highlighted the negative impacts that the residential schools had on individuals and communities. Abuse, separation from families, and the impacts on Aboriginal languages and culture are believed to have contributed to the family violence and substance abuse facing many First Nations, Métis and Inuit communities today.

The Indian Residential Schools Settlement Agreement, which came into effect on September 19, 2007, represents the culmination of the efforts of its signatories to work towards a fair, comprehensive and lasting resolution of the legacy of schools and the promotion of healing, education, truth and reconciliation and commemoration [Note 3].

Representing the consensus reached in the discussions between the Government of Canada, legal counsel for former students, the Churches, the Assembly of First Nations and other Aboriginal organizations, the out-of court settlement is recognized to be the largest class action settlement in Canadian history (A background study commissioned by Indian Residential Schools Resolution Canada estimated that there were some 80,000 former students still living in 2005) [Note 4].

According to the integrated RMAF-RBAF, approved in 2008 by Treasury Board, and soon to be reviewed as part of an evaluation framework exercise [Note 5], the Settlement agreement has two objectives:

• To resolve all outstanding claims as efficiently, effectively, humanely and compassionately as possible, while ensuring access to the appropriate level of health supports for Residential School claimants; and to
• To maintain and strive toward better collaborative working relations with Aboriginal organizations and their communities to promote and further develop programs that address the legacy of the Residential School system and encourage healing and reconciliation of former students, their families and communities.

Key Components of the Settlement Agreement

The Settlement Agreement includes both financial benefits for former students and non-compensatory initiatives directed to the benefit of former students and others, including family members and the general Canadian public, as described below:

Common Experience Payment

• This component is managed by INAC's Resolution and Individual Affairs Sector (RIAS). Common Experience Payments (CEP) are to be made upon application to all eligible former students. Payments are $10,000 for the first year, or part thereof, spent at an eligible school, plus $3,000 for each additional year, or part thereof.
• The Reconsideration Process, also managed by RIAS and launched in March 2008, is a second review of CEP applications to ensure that original decisions are accurate and appropriate.

Commemoration

• The Settlement Agreement identifies $20 million in funding for community, regional and national initiatives which are aimed at honouring and validating the healing and reconciliation of former students and their families. Eligible proponents are expected to include former students, their families, and/or other groups authorized by former students.

Independent Assessment Process

• The Independent Assessment Process (IAP) is an out of court, alternative dispute resolution process that resolves claims of abuse suffered at Indian Residential Schools. Former students who experienced sexual or serious physical abuse may be eligible for compensation through the IAP. The Indian Residential Schools Adjudication Secretariat (IRSAS) is the administrative body that supports the Chief Adjudicator in the implementation and administration of the IAP. IRSAS receives, assesses, and prepares claims for review and hearings before IAP Adjudicators.

Truth and Reconciliation Commission

• The Truth and Reconciliation Commission (TRC), was established as a federal department as of June 2008. It is intended to initiate and encourage reconciliation among former students, their families, communities, and all Canadians. In doing so, the TRC is expected to develop an accurate and public historic record of the Indian Residential School system and its impacts on former students.

Healing

An important part of the Agreement's purpose involves working closely with former students, their families, and communities in support of projects that promote healing. This is done primarily through programs offered by the Aboriginal Healing Foundation and Health Canada:

Healing Fund

• The Aboriginal Healing Foundation (AHF) delivers healing programs and services that address the experiences of former students, their families and communities. The Indian Residential Schools Settlement Agreement has set aside $125 million to the AHF for this work over five years (Under the provisions of the Settlement Agreement INAC will be undertaking an evaluation of the AHF during FY 2009/10).

Mental Health and Emotional Support Services

• The Indian Residential Schools Resolution Health Support Program provides emotional health and wellness supports to former students and their families. Health Canada has allocated a total of $97.5M to provide access to professional counselling and cultural and emotional support services (Elders / Resolution Health Support Workers).

Administrative Context

The Office of Indian Residential Schools Resolution of Canada was established as a department (by Order-in-Council) in 2001, to centralize and focus federal efforts to resolve claims associated with the operation of the former Indian Residential school system. Prior to that, a unit within Indian and Northern Affairs Canada was responsible for managing litigation, which was related to the operation of Indian Residential Schools.

On June 1, 2008, responsibility for the oversight of the timely and effective implementation of the Settlement Agreement was transferred to INAC's Resolution and Individual Affairs Sector. To this end, RIAS liaises with all other Program Interlocutors and the Courts to share information about efforts being made to implement the Settlement Agreement.

As discussed in the next section RIAS is responsible for the Advocacy and Public Information program, as well as for Commemoration, and is working with Service Canada for the delivery of Common Experience Payment process. The remaining components of the Settlement Agreement are administered by other government and non-governmental organizations:

• Truth and Reconciliation Commission – Truth and Reconciliation Commission
• Independent Assessment Process – Indian Residential Schools Adjudication Secretariat
• Mental Health and Emotional Support Services – Health Canada
• Healing Fund – Aboriginal Healing Foundation.

2.0 Program Profile

2.1 Overview

There are a number of parties engaged in disseminating information about the Indian Residential Schools Agreement. Each of the Settlement Agreement component programs have responsibilities for informing former students and/or the general Canadian population about their own respective activities. Additionally, the Courts have a role to play in disseminating information on the Settlement Agreement in general and INAC's Communication Branch, as part of its responsibilities for informing Canadians about INAC's programs and services, also plays a role in disseminating information.

As described below, the Advocacy and Public Information program, now managed by INAC's Resolution and Individual Affairs Sector, was designed to complement and broaden the communication activities of other stakeholders and parties.

2.2 Program Objectives and Expected Results

According to APIP's Terms and Conditions (2006), the program's objectives are to:

• Inform and encourage the sharing of diverse viewpoints on a range of Settlement issues, policies or programs that allow the Government to respond to the widest source of information possible and thus improve the quality of Departmental decisions;
• Identify where changes may be necessary to enhance service delivery;
• Ensure the Aboriginal community is informed of the benefits available through the Settlement Agreement; and to
• Help Canadians to understand the Settlement Agreement and the impact that the legacy of Indian Residential Schools has had on Aboriginal communities.

APIP's expected outcomes, also expressed in its Terms and Conditions (2006), are to:

• Final Outcome: Contribute to reconciliation between Residential School survivors and the Government of Canada;
• Intermediate Outcome: Improved community confidence and trust, access to quality well coordinated Residential School programs and services, and improved credibility for Residential School programs and services and the Government; and
• Expected Result: Former students of Residential schools will have complete access and equal participation in the Settlement Agreement.

2.3 Program Activities and Recipients

APIP is a proposal driven program which supports the following types of activities

• Workshops, conferences and gatherings on specific program-related topics or ranges of subject matters related to the Settlement Agreement;
• Studies that are published by third parties and publicly available; and,
• Efforts which bring recipients into contact with former students together in order to share information and discuss elements of the Settlement Agreement.

Eligible recipients include, among others, former students, Aboriginal organizations, public or private policy organizations, institutions or individuals with expertise on issues of interest, Tribal Councils, Aboriginal owned or controlled entities, organizations mandated to act on behalf of former students, and religious organizations and institutions.

Between April 2007 and November 2008, 42 contribution agreements were signed with 25 organizations, with an additional seven projects still in negotiation. Contributions ranged from $25,000 to $603,240 with the average being $200,866 (median $169,134, mode $25,000).

Contribution recipients to date have been predominantly Aboriginal organizations active at the regional and national levels. Projects (approved and in negotiation) include 19 national initiatives (9 focused on disseminating information on the settlement agreement and 12 on public education); and 30 regional initiatives (25 focused on disseminating information about the settlement agreement and 7 on public education) [Note 6].

The projects focus on a cross-section of different audiences, including First Nations, Inuit, and Métis, on- and off-reserve, rural and urban communities, and the general Canadian population. In fiscal year 2007-08, projects focused on disseminating information on the Common Experience Payment, while in 2008-09 projects are focusing on the Truth and Reconciliation Commission, the Independent Assessment Process and Commemoration.

2.4 Program Budget

APIP was allocated $10 million, under a 2-year allotment ending March 31, 2009. The program is seeking an extension for a further four years. The 2007-08 fiscal year funding allocation, of $6M, supported 27 initiatives, while the 2008-09 allocation of $4M has been committed to 15 projects, with an additional 7 projects still in negotiation.

Eligible expenditures include, in general terms, project manager wages; travel and accommodation for managers, staff and participants, elder honoraria, expert consultations and advice, instructional materials, and administration costs.[Note 7]

3.0 Evaluation Methodology

3.1 Evaluation Issues and Questions [Note 8]

This evaluation examined APIP's:

Relevance

• Does APIP continue to be aligned with Government of Canada and departmental priorities?
• Is there a continuing need for the program and what would the consequences be if APIP is not renewed?

Design and delivery

• To what degree are the program's logic model, performance measurement strategy and reporting requirements relevant, well articulated and understood, and do they support the measurement of expected outcomes?
• Is APIP approach and strategy effective?

Success and impact on end users

• To what extent is the program achieving its stated objectives and intended outcomes and are they consistent?
• Has APIP assisted former students to gain access to and participate equally in the benefits and programs provided under the Settlement Agreement?
• Has APIP helped to sensitize Canadians to the legacy of Indian Residential Schools?

Effectiveness, modifications and alternatives

• Are the most efficient means being used to achieve APIP objectives?
• Do gaps or duplication in service exist? Are clients receiving consistent information from different sources (i.e. INAC, APIP contribution recipients, Truth and Reconciliation Commission, Service Canada, etc.)?
• How could the program be improved, and what monitoring activity should occur to measure success?

3.2 Data Collection Strategy

The data collection strategy comprised multiple lines of evidence, including:

• Document and literature review, including internal documents, key documents related to the Settlement Agreement, as well as other settlement agreements related to other similar national/international experiences and/or advocacy and public information programs;
• A review of administrative data and information, including contribution agreements, project budgets, activity reports and financial data and statements;
• Secondary Data, including the Benchmark Survey on Public Awareness of the Indian Residential Schools Legacy, Environics Research Group, May 2008. This survey was jointly commissioned by IRSRC and the Truth and Reconciliation Commission to provide baseline data for performance measurement and evaluation purposes. The survey is a comprehensive quantitative survey of 1,500 Canadians, with a statistically representative oversample of 300 Aboriginal people living on and off reserve. The study offers a quantitative baseline of Canadians' general knowledge of Indian Residential Schools and the Settlement Agreement, with a particular focus on the long-term impact of this experience on former students, their families and their communities.
• Key informant interviews. A total of 23 interviews were conducted with INAC officials (n = 8), APIP officials (n = 4) other Settlement Agreement (Program Interlocutors) officials (n = 7), Contribution recipients (n = 3), and the IRSRC Elder-in-Residence;
• Electronic survey of program recipients (8 of the 25 Contribution Recipients participated in the survey which examined perspective on the evaluation issues, that is: relevance, design/delivery, success and impact on end users, and effectiveness); and
• Four project based case studies: Legacy of Hope: Our Stories Our Strengths; First Nations of Quebec and Labrador Health and Social Services Commission; Pauktuutit Inuit Women of Canada; and the B.C. based Indian Residential Schools Survivor Society.
  The four studies were selected to reflect a balance between national and regional projects. The cases were also selected for their potential to inform the evaluation's examination of delivery issues including linguistic diversity (official languages and Aboriginal languages); ethnic and cultural diversity (First Nations, Inuit or Métis); and delivery approach. The last included projects which focused on direct delivery to former students and others which targeted community leaders and service workers in a position to reach former students (e.g. frontline service providers, band councils), as well as projects focused on public education.

The evaluation Terms of Reference anticipated that a survey of survivors would also be conducted in order to examine the Program's performance. The survey was not conducted, in part because short time frame in which APIP has been operating, recognition of the sensitivities identified around Settlement agreement activities, as well as the identification of other sources of information on recipients and program uptake (e.g., including the 2008 baseline study conducted by Environics mentioned above, and program uptake data).

3.3 Research Limits

• As discussed under findings related to design and delivery, the evaluation was unable to assess progress towards the achievement of expected results or cost-effectiveness due to weaknesses in performance measurement.
• While steps were taken to overcome performance data gaps, a number of challenges, including scheduling issues, limited the scope of the evaluation's fieldwork. With respect to the case studies, for example, the evaluation design called for in-depth examination of program files, site visits and key informant interviews. However, site visits were made for only three of the four case studies and interviews with survivors could only be organized through two organizations. As such, the evaluation does not consider survivor perspectives identified in this report to be representative but indicative.
• The evaluation also faced challenges in attributing results to the Advocacy and Public Information program, partially as a result of performance measurement issues, and partially because of the number of parties who are also engaged in disseminating information regarding the Settlement Agreement.

4.0 Findings: Relevance

The questions addressed for this evaluation issue were:

• Does APIP continue to be aligned with Government of Canada and departmental priorities?
• Is there a continuing need for APIP and what would the consequences be if APIP is not renewed?

4.1 Key Findings

The implementation of the Indian Residential Schools Settlement Agreement remains a federal and departmental priority.

The current government took significant steps to re-assert the priority placed by the government on recognizing the importance of the legacy of the Residential Schools and the implementation of the Settlement Agreement when, in June 2008, the Prime Minister issued a Statement of Apology to former students of Indian Residential Schools, on behalf of the Government of Canada [Note 9].

There is a continuous need for communications activities which will enhance uptake of the Settlement Agreement programs and increase awareness about the Residential experience.

Settlement Agreement Information
APIP was intended to support the initial launch of Settlement Agreement's financial and non-financial benefit programs. However, the Program has not been able to fulfill its mandate because, while the Settlement Agreement came into effect September 19, 2007, some activities have been delayed and some not yet launched, as shown below:

According to Program officials, priority has been placed on disseminating information about activities which were expected to be delivered or initiated during each of the two fiscal years APIP has been operating. In 2007-08, for example, projects focused on the Common Experience Payment, while in 2008-09, priority was placed on the Truth and Reconciliation Commission and Commemoration (neither of which in fact have yet initiated activities).

Regardless of delays relating to implementations, many key informants interviewed stated that the need for information on the Settlement Agreement and its legacy has not been completely met. This appreciation is supported by evidence from an analysis of departmental data and secondary sources, including surveys conducted before and after the Government of Canada's statement of apology.

The evaluation's analysis of departmental data, circa January 2009, found, for example, that while some regions are reporting Common Experience Payment applications far in excess of estimated students, others are reporting far less than would be expected (Appendix A). While it was beyond the scope of this evaluation to clearly identify the reasons behind these patterns, it does suggest that not all potential applicants are receiving information, or appropriate information, or have chosen not to participate in the process.

Supporting the possibility that access to information is likely one factor is the finding that just prior to the Prime Minister's statement of apology [Note 10], about 20% of the on-reserve population and more than one-third of the off-reserve First Nation population did not know about the common experience payment and about two-thirds of the Aboriginal population overall did not know of upcoming commemoration activities as noted below. If eligible recipients - former students, families, communities - are not aware of the supports available to them, they will not access them or otherwise participate fully in the resolution process.

Source: Environics Research Group, (2008). The general population sample was stratified regionally to provide for reliable analysis by region (Atlantic Canada, Quebec, Ontario, Western provinces, and the North). Oversamples of the Aboriginal population were distributed proportionately across the country.

Public Education
In terms of public education requirements, the 2008 National Benchmark Survey also revealed that overall, half (51%) of Canadians report that they had heard or read something on the subject of Indian Residential Schools, as compared to eight in ten Aboriginal Canadians. (Environics Research Group, 2008: 13):

Among Canadians generally, awareness of Indian residential schools is highest in the North (86%) and in Western Canada (72%). About half of Ontarians (49%) and Atlantic Canadians (45%) are aware of them, while only 27% of Quebecers could recall seeing or hearing anything about these schools [Note 11].

Figure 1 Awareness of Indian residential schools [Note 12] (May 2008)

Note: Amounts noted in figure above are percentages of overall response rate of respective sample population.

As seen in Figure 1, overall, half (51%) of Canadians report that they have heard or read something on the subject of Indian residential schools. This proportion rises to 80% among Aboriginal Canadians living on reserve and 79% living off reserve.

Given the extensive media coverage of the Prime Minister's Statement of Apology regarding Indian Residential Schools, on June 11, 2008, it is fair to say that the National Benchmark Study may understate the level of awareness of the general Canadian population. An Angus Reid Strategies poll, released August 11, 2008, revealed that 67% of respondents strongly or moderately agreed with the Apology. Conversely, 17% strongly or moderately disagreed while 16% were not sure [Note 13]. While the survey question does not directly deal with perceptions on the Residential Schools legacy, it does suggest that public education and awareness may have increased some after the statement of apology but there are still gaps in information and awareness.

4.2 Summary / Conclusions

The implementation of the Indian Residential Schools Settlement Agreement remains a federal and departmental priority, and with current delays in implementation of the Settlement Agreement, there continues to be a need for communication activities.

Among these needs, the evidence suggests that former students continue to require information on ongoing initiatives and will require information on activities yet to be launched. According to program recipients and secondary survey evidence, former students are fairly well informed about the financial compensation related to the Settlement Agreement, however they are predominantly unaware of other benefits available. There also continues to be a need for public education to sensitize the general Canadian population on the legacy of the Indian Residential Schools experience on Aboriginal communities.

5.0 Findings: Design and Delivery

The questions addressed for this evaluation issue were:

• To what degree are the program's logic model, performance measurement strategy, reporting structure and requirements relevant, well articulated and understood, and do they support the measurement of expected outcomes?
• Is APIP's approach and strategy effective?

5.1 Key Findings

Performance Measurement

APIP does not have a clearly articulated results based management or performance measurement framework.

Gaps identified through the evaluation research include:

• There is a lack of clarity in expectations for the program, and the links between objectives and expected outcomes are similarly weak. APIP's Terms and Conditions (2006) and the later IRSRC Integrated RMAF/RBAF (2008), for example, outline two different sets of expected results and program objectives. (This evaluation draws upon the version appearing in the Terms and Conditions, given the authority of this document and because they were distributed with APIP's calls for proposals.)

• Key terms such as 'advocacy' and 'reconciliation' are not well defined or clearly understood by stakeholders (To note, the clarity of terms associated with the Settlement Agreement will be reviewed as part of the development of an upcoming evaluation framework exercise to be conducted by EPMRB for programs administered by RIAS).
• No documented evidence was found to suggest that the Program is drawing upon available statistical data (at the time of inception, a commissioned study on estimated numbers of former students, and presently uptake data. (At this point, Environic's 2008 National Benchmark Survey is also available to inform planning and performance measurement).
• Performance measurement indicators, as expressed in IRSRC's integrated RMAF/RBAF are not outcome oriented and are not well linked to expected results [Note 14]. Where data is being collected, it does not appear to be routinely disaggregated by age, gender (in line with INAC's Gender Based Analysis Policy) or other demographic characteristics. This appears to be a particularly problematic gap in light of the fact that background documents indicate that priority is being placed on addressing needs of the elderly. An example of this can be found in the Reconsideration process. In efforts to ensure fairness and transparency while balancing the urgency associated with the most elderly, reconsideration requests are processed based on the following priority: age (where applicants were 65 or older as of May 30, 2005); and in the order in which CEP applications were received.
• Current APIP application and reporting requirements are not set up in a manner to facilitate reporting on results or performance measurement. The February 2008 APIP Application Form, for example, leaves proponents to identify expected results and the means by which success would be measured, without reference to any indicators or measures that INAC might require to monitor and measure results.
• Current selection criteria, as expressed in proposals, call letters, and APIP website, focus on themes to be addressed by projects, but only identifies target audiences in general terms.

Delivery Approach

APIP's strategy was found to be appropriate given the sensitivity of the information to be shared.

The Settlement Agreement takes the sensitivity of residential school issues seriously. All Settlement Agreement program websites, for example, have a disclaimer advising that the subject matter may be disturbing to some visitors and to Survivors of the residential school system and offers information for individuals to seek support if necessary (i.e. 1-800 telephone number).

The Official Settlement Notice Plan on the Indian Residential Schools Settlement Agreement observed that in-person dissemination of information was critical to reach as many former students and their families as possible:

"These grass-root efforts, to be designed chiefly by the AFN and various Inuit organizations, and possibly others, will provide additional notice exposure beyond that which will be provided through mailings and paid measurable media, and will allow for face-to-face explanations of notices and answers to basic questions regarding the Settlement and Class members' rights and options" [Note 15].

In recognition of the sensitivities involved, APIP opted to provide support to existing, credible, organizations, primarily Aboriginal, to approach former students, family and community members and to encourage face-to-face communications (e.g., as noted in Section 3, fundable activities include workshops, conferences and gatherings as well as other activities to support face-to-face meetings with former students).

A significant degree of evidence was found to support APIP's approach, for example:

• Case studies and key informant interviews indicated that face-to-face communications, from trusted sources, were conducted and are crucial for communication within Aboriginal communities. Survivors interviewed for the case studies commended the face-to-face interaction and workshop as a communication strategy to disseminate Settlement Agreement information.
• The literature and document review along with key informant interviews and case studies indicates that information disseminated related to the Indian Residential Schools Settlement Agreement can trigger an emotional response and at times opens up trauma for former students.
• A report by the Law Commission of Canada, released March 2000, entitled Restoring Dignity: Responding to Child Abuse in Canadian Institutions outlined recommendations in relation to operations of redress programs which include outreach activities. It stated that "consideration must be given to contacting former residents in a manner that is least likely to cause harm to them" [Note 16].
• In a paper prepared for the Solicitor General of Canada, entitled Victims Of Crime and the Justice System In Ontario: An Issues Paper, it was noted that victim advocacy groups were supportive of proactive contact. These groups recognized that such activity may upset some individuals (a relatively small proportion), but mostly eligible claimants would welcome, even expect information [Note 17].
• The literature review also revealed that "the diversity of Aboriginal communities makes it impossible to devise one approach which will meet the needs of all people" [Note 18]. Additionally, it was noted that non-Aboriginal organizations "involve the imposition of an infrastructure inconsistent with Aboriginal culture" [Note 19] and that the most effective activities support and empower communities to meet unique needs of individuals.

The evidence also suggests that this strategy was an appropriate response to the Program's two year time frame and the urgency of disseminating information about the Settlement Agreement.

Case studies, key informant interviews, document and administrative data review revealed the engagement of former students, their families and communities in activities. By entering into Contribution Agreements with primarily regional and national Aboriginal organizations, the Advocacy and Public Information program benefited from their experience and knowledge of cultural and community context. According to Survivors interviewed, this approach is both credible and effective in achieving results.

In light of the projects initial two year time frame, Program officials chose to work primarily with Aboriginal organizations with existing capacity in order to achieve efficiencies and quickly distribute funds for communications of the Settlement Agreement.

The one challenge which the evaluation found associated with APIP's approach is that there appears to be no contingency plans in place for cases in which no capable or willing organizations have been identified. At the time of the evaluation, there continue to be seven projects under negotiation, and presumably, target audiences without access to the services provided through the program.

5.2 Summary / Conclusions

The lack of clarity in the program's expected results and performance measurement gaps severely inhibit the Program's capacity to monitor performance, measure the achievement of expected results, or identify issues or factors which may differentially affect access to information and participation in benefits from the Settlement Agreement. These gaps have also severely limited the evaluation's capacity to assess expected results and cost-effectiveness.

APIP's strategy of engaging credible and trusted Aboriginal organizations was found to be appropriate given the sensitivity of the information to be shared. It is also an appropriate response to the urgency of disseminating information about the Settlement Agreement. One challenge, however, is that no contingency plans are place to inform individuals and communities when capable and willing organizations at the community level cannot be identified in a timely manner.

6.0 Findings: Success and Impacts

The questions addressed for this evaluation issue were:

• To what extent is the program achieving its stated objectives and intended outcomes and are they consistent?
• Has APIP assisted former Residential School students to gain access to and equal participation in the IRSSA?
• Has APIP helped to sensitize Canadians to the impacts of the Indian Residential Schools experience on Aboriginal communities?

6.1 Key Findings

The evaluation was unable to assess to which degree the Program's expected results were being achieved.

As discussed previously, the following variables have severely limited the evaluation's ability to assess achievement of expected results: the lack of a program specific logic model, significant gaps in baseline data, performance information and the absence of targets.

Another factor with significant limitations in the assessment of the evaluation was the short timeframe that has lapsed since APIP began implementation. The first set of contribution agreements were only signed in April 2007. Furthermore, the potential to assess outcomes was limited by the fact that due to delays in Settlement Agreement activities APIP has not yet disseminated information on all planned Settlement Agreement activities (see Section 4.1).

The evaluation did identify a number of promising practices towards meeting the needs of former students.

• The Indian Residential School Survivors Society (IRSSS) project involves a province wide dissemination strategy that focuses on reaching leaders and others capable of connecting with survivors on an ongoing basis. To that end IRSSS facilitated 72 workshops with former students in communities across British Columbia.
• IRSSS also facilitated training for trainers to help entrench knowledge and capacity in local communities, who offer ongoing support to survivors. Informants noted that the approach used by IRSSS' initiatives represent a solid move toward understanding, healing and communication, and lays the groundwork for reconciliation. This approach also leads to a position in which Aboriginal communities and the Government of Canada will be able to move forward.
• In 2007, the Assembly of Manitoba Chiefs facilitated a national conference and workshop. It brought together residential school survivors, their descendents, political leaders, management and frontline workers from Federal, Provincial/Territorial and First Nations governments. The conference brought together 1,600 participants who contributed to a national strategy to minimize possible negative impacts and maximize the positive effects of the Indian Residential Schools Settlement Agreement [Note 20].

APIP has made some contribution toward sensitizing Canadians.

The Legacy of Hope Foundation (LHF) is a national, charitable organization whose purpose is to educate and create awareness and understanding about the legacy of residential schools and the effects and intergenerational impacts on First Nations, Inuit and Métis peoples and to continue to support the ongoing healing processes [Note 21]. The following two LHF projects funded by APIP are examples of some promising practices:

• "Where are the Children?" is a traveling exhibit on the Indian Residential Schools experience and legacy. To date, 200,000 Aboriginal and non-Aboriginal people have participated in the exhibit.
• "Our stories, Our Strength" is a collection of former students' stories to be published in a number of media. APIP supported the production of a seven-minute DVD. The case study revealed that to date 500 copies have been disseminated. The process allowed for the collection of 560 stories, in 21 communities across the country. In the case study, it was noted that for many survivors, this process allowed them to change their lives by breaking the isolation and giving them an opportunity to pursue therapy. The case study also revealed that individuals felt that sharing their stories was liberating and helped them to feel better.

Overall, the evidence suggests that, while there have been notable achievements in reaching out to the general Canadian population, APIP's main focus has been on former students, their families and their communities.

While national projects tended to focus on public information, regional level projects have tended to focus on disseminating information regarding the provisions of the Settlement Agreement, as well as its associated benefits and programs. Bearing in mind that projects may undertake activities related to the dissemination of information and public education, 12 of 19 national level projects focused on public education, and 9 on disseminating information to potential beneficiaries or service agencies. 25 of the 30 regional projects focussed on the Settlement Agreement; only 7 focused on public education.

The evidence is mixed on the extent to which APIP has assisted former students to gain access to and equal participation in the Settlement Agreement programs.

Table 4, below, uses Departmental estimates of Residential School Attendees as a proxy for baseline data on information gaps and needs by region as of September 2007 (the implementation date of the Settlement Agreement). It is important to note that this analysis does not aim to draw a correlation between APIP funding and Settlement Program uptake, which is partially due to challenges with attribution given other parties involved, and gaps in performance measurement. Rather the analysis is intended to demonstrate the potential of using such information to identify where the information needs might be most required and to compare those needs with APIP expenditures.

In general, APIP project funding seems to be proportional to regional needs (comparison of columns B and F). In some areas, where the funding was somewhat less, such as Alberta and Saskatchewan, this did not seem to have a bearing on uptake. 105% and 115% of Estimated Residential School Attendees in these regions were issued a Common Experience Payment, exceeding estimations. This suggests that information and supports in these regions are helping former students access Common Experience Payments.

Quebec is the only region that received proportionately less funding compared to estimated baseline need, while percentage of payments is significantly less than initial estimations. Within the Quebec case study, which was conducted for the evaluation, one challenge that was identified was accessibility to information in French, which may offer a partial explanation for the results achieved.

(Source: See Appendix A)

Based on further analysis of departmental data (see Appendix A), the median of payments made on applications was 80% while the average was 75% (see Appendix A). This offers an illustration that on average, approximately one in five applicants were either deemed ineligible for payment or their applications were put forward for reconsideration. This offers an indication which information that was available to individuals related to accessing programs, was insufficient for effective participation and/or that people were unclear about the parameters of the Settlement Agreement (Residential Schools vs. Day Schools).

It is worth noting that the North and Saskatchewan had significantly higher rates of payments issued in relation to estimations, 142% and 115% respectively. his suggests that the mechanisms for disseminating information within these regions may have contributed to the overall success of reaching intended audiences. Again these results cannot solely be attributed to APIP, but APIP's efforts may have some implications for understanding regional variances. One factor which may have contributed to high uptake in the North is the attention paid by the recipient organization on the importance of addressing linguistic diversity. The Dene Nation, for example, ensured that conference communications were conducted in five local languages.

The evaluation evidence collected through key informant interviews, case studies and the document review also suggests that clarity and quality of information provided may be an issue affecting uptake and participation rates. Steps were put in place at the onset of the implementation of the Settlement Agreement to ensure consistency in the information imparted by differing parties [Note 22]. However, key informant interviews and case studies indicated repeated cases of individuals who did not understand why the school they attended was not included in the Settlement Agreement (i.e. day schools) and that application forms were difficult to complete. This raises questions about the clarity of approved communications materials for intended audiences and whether some revisions may be necessary.

6.2 Summary / Conclusions

While a number of promising practices were identified, the evaluation was unable to assess the degree to which the Program's expected results are being achieved due to the weaknesses in performance measurement, and the short time frame in which the program has been operating.

According to an analysis of departmental data, APIP project funding seems to be distributed in proportion to regional needs, but no simple correlation was found between funding and uptake. The evidence suggests that future initiatives should take additional steps to ensure the clarity of the information disseminated and ensure mechanisms are in place to identify communication related issues affecting applications.

7.0 Findings: Effectiveness

The evaluation questions addressed for this issue were:

• Are the most efficient means being used to achieve APIP objectives?
• Do either gaps or duplication in service exist between INAC's Resolution and Individual Affairs Sector and/or partners? Are clients receiving consistent information from different sources (i.e. INAC, APIP contribution recipients, Truth and Reconciliation Commission, Service Canada, etc.)?

7.1 Key Findings

Gaps in performance data and the short-time frame in which projects have operated negated the potential to evaluate the program's cost-effectiveness.

As discussed previously, the lack of a program specific logic model, significant gaps in baseline data, performance information and the absence of targets severely limited the evaluation's ability to assess key evaluation questions, including, in this case, cost effectiveness.

Efforts are being taken to reduce overlaps and gaps and to enhance coordination among Program Interlocutors and Contribution Recipients to strengthen implementation of Settlement Agreement.

The evidence suggests a promising informal division of labour between Settlement Agreement programs and APIP supported organizations has emerged, as has an informal network of these parties. According to Key Informants and case studies, Contribution Recipients are offering former students general information about the Settlement Agreement and related programs, and Program Interlocutors are responding to more technical aspects related to their areas of responsibility.

Additionally, it was noted that the practice of having Program Interlocutors attend APIP supported workshops is proving effective, both in transmitting information and as a forum for raising issues and concerns. Gaps in performance data do not allow an assessment of the degree to which the respective programs have been able to act on information provided.

An example of this is one concern that was raised repeatedly in the key informant interviews and case studies. Here it was noted that the multiple 1-800 telephone numbers related to different programs and supports served to complicate matters as stakeholders felt it was hard to discern which one to contact for what purpose. Informants reported that when people did call they were often forwarded from one to another, yet this still did not help to answer their questions. Specific examples of what questions went unanswered were not provided). Additionally, a few key informants noted concerns about the cultural sensitivity of telephone operators in responding to former students.

Program officials admit that they have facilitated teleconferences which bring together Contribution Recipients to share information, but that the calls do not take place on a consistent basis. Most recently, however, they have established an informal network of Contribution Recipients and Program Interlocutors in the interest of sharing information amongst parties and as a means of fostering two way communications between recipients and interlocutors.

This networking approach is consistent with the RIAS' broader partnership activities that support horizontal cooperation, such as the Community Impacts Working Group. RIAS facilitates discussions among this group, which brings together other government departments and key stakeholders to share information about local, regional and national efforts to implement the Settlement Agreement. At present this group does not convene frequently, however RIAS has made a recent commitment to utilize this resource more effectively. This could present an opportunity to serve as a mechanism to help coordinate communications activities to support consistency of information dissemination and to identify areas for improvement in service delivery and avoid duplication of efforts.

As useful as the above efforts may be for reducing the potential of overlap and for harmonizing communication activities, the current Settlement Agreement Structure lends itself to potential overlaps which may increase as differing components come on line. As earlier indicated, while APIP is the only program currently working with Aboriginal organizations for the purpose of communications:

• Other Settlement components, including the Independent Assessment Process and the Truth and Reconciliation Commission, disseminate information on their respective activities;
• In addition to APIP, INAC's Communications Branch and the Courts disseminate information on the entire Settlement Agreement; and
• Both APIP and the Commemoration Program have similar public education objectives and contribution funding to support communication activities.

7.2 Summary / Conclusions

With more Settlement Components coming on-line, and some potential opportunities for overlap being noted now is the time to ensure the roles and responsibilities are clear and complementary, and structured or streamlined in such a manner so as to best facilitate uptake.

As in other areas of investigation for this evaluation, efforts to examine cost-effectiveness were negated by the lack of a program specific logic model, significant gaps in baseline data, performance information as well as the absence of targets.

8.0 Conclusions and Recommendations

The evaluation supports the following conclusions on relevance, design and delivery, success and effectiveness.

8.1 Conclusions

The implementation of the Indian Residential Schools Settlement Agreement remains a federal and departmental priority. With current delays in the implementation of the Settlement Agreement, there continues to be a need for communication activities directed to survivors, their families and communities and the broader Canadian population.

APIP's strategy of engaging credible and trusted Aboriginal organizations was found to be appropriate given the sensitivity of the information to be shared. It is also an appropriate response to the urgency of disseminating information about the Settlement Agreement. One challenge, however, is that no evidence emerged over the course of the evaluation that a formal contingency plan is in place to inform individuals and communities when capable and willing organizations at the community level cannot be identified in a timely manner.

The lack of clarity in the program's expected results and performance measurement gaps severely inhibit the Program's capacity to monitor performance, measure the achievement of expected results, or identify issues or factors which may differentially affect access to information to and participation in benefits from the Settlement Agreement. These gaps have also severely limited the evaluation's capacity to assess expected results and cost-effectiveness.

In addition to performance measurement gaps, the short time frame in which APIP has been operating also limited the potential for assessing impacts at this time. A number of promising practices were identified in terms of maximizing the reach of communication activities. Moreover, according to an analysis of departmental data, APIP project funding seems to be distributed in proportion to regional needs, but no strong correlation was found between funding and uptake. The evidence suggests that future initiatives should take additional steps to ensure clarity in information disseminated and ensure mechanisms are in place to identify communication related issues affecting applications.

Efforts are being taken to reduce overlaps and gaps and to enhance coordination in order to enhance the results of Residential School related communications. With more Settlement Components coming on-line, and some potential opportunities for overlap being noted, now is the time to ensure the roles and responsibilities are clear and complementary, and structured or streamlined in such a manner so as to best facilitate uptake.

8.2 Recommendations

1. In light of the revised implementation schedule for various Settlement Components, INAC should consider continuing support for communication activities, based in part on the APIP model, to assist in informing former students and Canadians about the Settlement Agreement;
2. Clarify roles and responsibilities of the various parties involved in disseminating information about the Settlement Agreement so as to ensure cost-effectiveness in the delivery of information and reduce the potential for duplication and overlaps over time; and
3. Depending on the approach adopted, develop a results based management tools and performance measurement strategy, that is mindful of reporting burden, yet includes:
   
   
   • a program specific logic model demonstrating links to the Settlement Agreement objectives and INAC's program activity architecture;
   • a needs assessment exercise which draws upon existing data (e.g., baseline study, program uptake rates) disaggregated by location, age and gender;
   • performance measurement indicators and targets;
   • formalized recipient selection and assessment criteria which are clearly aligned with program objectives (particularly with respect to reach)
   • results-based reporting requirements; and
   • a client survey to be applied for all activities undertaken (i.e. clarity and utility of information provided, change in understanding, and ability to act based on information provided).

Appendix A:

Comparative Analysis of Estimate Residential School Attendees and Common Experience Payment

Appendix B: Management Response / Action Plan

Footnotes

1. This component of the Settlement Agreement is delivered by Health Canada through the Indian Residential Schools Resolution Health Support Program. (return to source paragraph)
2. Key sources: IRSRC, (2008: 5), Integrated Results-based Management and Accountability Framework and Risk-Based Audit Framework (RMAF/RBAF): IRSRC, Departmental Performance Report, (2007 – 2008), and Statistics Canada, (2003: 22), Aboriginal Peoples Survey 2001 - Initial findings: Well-being of the non-reserve Aboriginal population, (Catalogue 89-589). (return to source paragraph)
3. Indian Residential Schools Settlement Agreement, Preamble, B and C, (2007: 6). (return to source paragraph)
4. See: IRSRC, Departmental Performance Report (2007-08); Siggner and Associates Inc., Estimating the Residential School Attendee Population – For the Years 2001, 2005 and 2006. (return to source paragraph)
5. EPMRB expects to begin work the first quarter of 2009 on the development of an Evaluation Framework for Settlement Agreement programs administered by RIAS (the Common Experience Payment, the Independent Assessment Process, and the Commemoration Program). (return to source paragraph)
6. The number of projects focused on disseminating information on the Settlement Agreement and public education exceeds the total number of projects because some projects address both types of activities. (return to source paragraph)
7. Indian and Northern Affairs Canada (return to source paragraph)
8. The ordering and wording of the questions have been slightly revised from the Terms of Reference approved by the Audit and Evaluation Committee in June 2008. To note: the final question in the list above, related to program improvements, is largely addressed in the evaluation conclusions and recommendations (Section 8). (return to source paragraph)
9. 39th Parliament, 2nd Session, Edited Hansard, Number 110, June 2008. (return to source paragraph)
10. Environics Research Group, (2008), National Benchmark Survey, May 2008. The survey measured the level of awareness of the general Canadian population,, with an oversampling of Aboriginal people living on and off reserve, on the Settlement Agreement and the legacy of Indian Residential Schools. (return to source paragraph)
11. Ibid, page 14. (return to source paragraph)
12. Ibid. (return to source paragraph)
13. Angus Reid Strategies, (2008: 3), Canadians Agree with Prime Minister's Apology to Aboriginal Community, August 2008. (return to source paragraph)
14. id="note14">The three indicators noted in the IRSRC Integrated RMAF/RBAF are: # conferences, community workshops, information sessions attended and presentations made by IRSRC staff directly to former students, their families, and front-line support workers; feedback from Community Impacts Working Group [Note this acronym is not defined or mentioned elsewhere in the document); and of communities visited by project staff. (return to source paragraph)
15. Hilsoft Notifications, (2006: 53), In re Residential Schools Class Action Litigation Settlement Notice Plan, February 2006. This plan, approved as part of the Indian Residential Settlement Agreement, was designed with input from Aboriginal people and groups, lawyers for the parties, the Government and with direction from the Courts. The focus of the Plan was on opt-out (of the Settlement Agreement) and advance payments and had objectives in this context to notify the greatest number of former residential school students and their family members, and provide them with opportunities to see, read, or hear notice, understand their rights, and respond if they choose. (return to source paragraph)
16. Fred Kaufman, (2002: 21)l Searching for Justice: An Independent Review off Nova Scotia's Response to Reports of Institutional Abuse, Chapter XVII - Report of the Law Commission of Canada, Province of Nova Scotia, January 2002. (return to source paragraph)
17. Solicitor General Canada, (1994: 27), Victims Of Crime and the Justice System In Ontario: An Issues Paper. (return to source paragraph)
18. Ibid. p. 31. (return to source paragraph)
19. Ibid. (return to source paragraph)
20. Indian Residential School Survivors, (2007: 2), National Conference and Workshops - Final Report: Preparing Survivors and Communities to Move Forward, Winnipeg, 2007. (return to source paragraph)
21. Legacy of Hope. (return to source paragraph)
22. The Document Review indicated that Program officials advised contribution agreement holders upfront that communications were to be consistent with the formal court Notice Plan that all communications materials must be consistent with that Notice process that was approved by the National Certification Committee. All workshops conducted with former students of IRS were to offer products approved by the NCC and use the official court notices. (return to source paragraph)