Joint Evaluation of the On Reserve Income Assistance Reform

Introduction

The evaluation of the On-Reserve Income Assistance Reform initiative was conducted in accordance with the Treasury Board's 2009 Policy on Evaluation. The evaluation examined Income Assistance Reform's relevance, performance (effectiveness, efficiency and economy), and design and delivery from its inception in April 2013 to July 2015. One key objective of the evaluation was to produce timely findings and conclusions intended to inform the possible renewal of Income Assistance Reform.

The evaluation was conducted jointly by the Evaluation, Performance Measurement and Review Branch at Indigenous and Northern Affairs Canada (INAC) and the Evaluation Directorate at Employment and Social Development Canada (ESDC).

Background

Income Assistance Reform is an interdepartmental initiative announced in Budget 2013,^{Footnote 1} whereby INAC and ESDC collaborate to implement a continuum of services to address barriers to employability of Income Assistance clients aged 18 to 24 who are expected to be deemed as employable within one year.

INAC administers the Enhanced Service Delivery component. Its delivery is ensured by First Nations organizations such as band councils and tribal councils. First Nation communities use a case management approach to identify clients' individual barriers to employment, and establishing an action plan to address these barriers through available programs and services. It is focusing on activity areas such as: pre-employment training on basic literacy skills; life-skills training; essential skills training; high school diploma completion/equivalency; psychosocial interventions; and financial incentives for participation in mandatory provincial or territorial intervention.

ESDC administers the First Nations Job Fund (FNJF) component. The FNJF is delivered by Aboriginal Skills and Employment Training Strategy service network. FNJF service provider's role is to provide skills development training to secure employment or return to school, as well as other types of active measures such as: skills assessments; personalized training; coaching; and other supports for young Income Assistance clients on-reserve.

Enhanced Service Delivery and the FNJF services providers are required to develop a Client Referral Strategy, which establish service standards between both parties and outline how the organizations will work together to serve Income Assistance clients.

It is estimated that 25,547 youth on-reserve between the ages of 18-24 are currently in receipt of Income Assistance. By 2016-2017, Income Assistance Reform intends to reduce the number of clients on Income Assistance by 4,277 individuals. Over a period of four years (2013-14 to 2016-17), INAC is expected to allocate $132.5 million for the Enhanced Service Delivery^{Footnote 2} and ESDC will allocate $108.6 million for the FNJF.

The ultimate outcomes of Income Assistance Reform are: First Nation men, women and children are active participants in social development within their communities; and First Nations Income Assistance clients on-reserve are employed and integrated into the labour market.

Methodology

The methodology used to conduct this evaluation included: a document and file review; a literature review; data analysis; 20 key informant interviews with INAC and ESDC program staff; and seven site/community visits.

Findings and Conclusions

This section summarizes the evaluation findings related to the relevance, design and delivery and performance of the Income Assistance Reform.The program's assessment of performance is based on the achievement of its immediate outcomes. Once the program has sufficiently matured and employment income data becomes available, an impact analysis will determine whether the Income Assistance Reform can be attributed with sustainably reducing the number of youth relying on income assistance. Future evaluation work will also be better positioned to examine the Income Assistance Reform's cost-effectiveness.

Areas for Potential Improvement

In general, a few areas were identified for potential adjustment to the Enhanced Service Delivery design that includes the following:

• The length of the Enhanced Service Delivery intervention may not be sufficient (currently set at a maximum of six months) to address all Income Assistance clients' needs with respect to employment readiness.
• Currently, there are no clear regulations for program administrators or clients on return to or resuming programming once they have left Income Assistance Reform activities.
• Extending the age of accessibility to 30 years of age may benefit more youth.
• Examining program participation incentives to ensure they do not motivate clients to apply for Income Assistance.
• Examining the consistent application of disincentives to clients who are not meeting the targets established in their action plans.
• Reducing the 50:1 caseworker caseload so that Enhanced Service Delivery caseworkers may provide the personalized assistance clients require to complete their action plans.

Recommendations

Most of the recommendations in this report stem from aspects of data collection that are intended to improve ongoing performance measurement and to support future policy, research, and evaluation work. There were concerns over the completeness and reliability of some of the data. For example, the socio-demographic and intervention data collected by the Enhanced Service Delivery was insufficient to adequately describe participants and the types of interventions they participated in. An examination of the data collection strategy by both departments is needed to more efficiently capture and link participant data to ensure effective client monitoring and analysis of outcomes and impacts.

Recommendation 1: It is recommended that INAC ensure that data collection instruments and processes used by service providers consistently and accurately capture adequate socio-demographic and program outcome information on clients.

Recommendation 2: It is recommended that INAC and ESDC collaborate to explore ways to more efficiently capture and link information collected by each department to allow for strengthened client monitoring and analysis of outcomes and impacts of the Income Assistance Reform.

FNJF client data suggested that just over one-quarter of clients (28 percent) were recorded as not being in receipt of Income Assistance. Further investigations suggest that this can be attributed to a data entry error. For example, some FNJF caseworkers were inadvertently recording clients as not being on Income Assistance after having found employment.

Recommendation 3: It is recommended that ESDC communicate to all FNJF service providers the data capture protocols that enable the accurate and consistent recording of the client status of being on Income Assistance.

Finally, the evaluation identified some potential to improve efficiency, such as examining the feasibility of multi-year funding agreements for Enhanced Service Delivery providers. For example, late notification of 2014-2015 funding led to staff layoffs and pause in client interventions in some delivery sites. This inevitably caused delayed referrals to the FNJF. Funding the Enhanced Service Delivery on a multi-year basis would reduce service interruptions and improve delivery of the Income Assistance Reform.

Recommendation 4: It is recommended that INAC explore the feasibility of introducing multi-year funding agreements for the Enhanced Service Delivery.

1. Management Response

The Income Support Directorate at Indigenous and Northern Affairs Canada (INAC) and the Aboriginal Affairs Directorate at Employment and Social Development Canada (ESDC) accept the recommendations that were presented in the Joint Evaluation of the On-Reserve Income Assistance Reform initiative and will endeavour to implement solutions to the issues raised therein.

Recommendation 1: It is recommended that INAC ensure that data collection instruments and processes used by service providers consistently and accurately capture adequate sociodemographic and program outcome information on clients.

Over the next fiscal year, INAC will implement a data collection initiative to improve information on clients so as to inform future policy decisions. This initiative will take place between January 2016 and January 2017, and will involve 60 Enhanced Service Delivery providers. The objective is to collect additional sociodemographic information, such as client barriers to employment. This client-level information will capture the path of individuals through the case management and referral system in order to better identify challenges faced by clients in accessing the program, transitioning to the work force, and maintaining employment. As part of its day-to-day program management, INAC will also continue to support providers in using the current Data Collection Instrument, which provides aggregated information on: educational attainment; family composition; age; sex; participation in active measures; exit to employment; and education.

In view of a desire to reduce the reporting burden for reserve communities, major revision of the Income Assistance program Data Collection Instrument is planned for 2016-2017. Additionally, revision of the joint Performance Measurement Strategy is planned for the same year. Both revised items will be implemented in 2017-2018. The revision of the latter document will help ensure that data fields and indicators are aligned with relevant outcomes and departmental policy on reporting.

Options such as the development of an automated case management system will be considered with a view to improving Income Assistance program information, data collection and data reporting processes. Such options will focus on increasing the capacity of service providers to effectively manage client caseloads.

Recommendation 2: It is recommended that INAC and ESDC collaborate to explore ways to more efficiently capture and link information collected by each department to allow for strengthened client monitoring and analysis of outcomes and impacts of Income Assistance Reform.

To-date, INAC and ESDC have been working to ensure that timely and relevant performance information is available to support effective oversight and accountability for the Income Assistance Reform initiative. The evaluation has been helpful in identifying vulnerabilities and data gaps that should be addressed if the departments are to demonstrate results and the impact of their combined efforts over the long-term. INAC and ESDC have a Memorandum of Understanding for Information Sharing which focuses on sharing aggregate level data for implementation purposes, including project approval, results monitoring and evaluation. In addition, the Program Review and Advisory Committee approved a pilot project involving a select number of Enhanced Service Delivery providers and focused on improved data collection and sharing practices.

INAC and ESDC, through the Income Assistance Reform's interdepartmental Performance Measurement Working Group, will collaborate to identify short-term options to improve client data collection and analysis in support of improving the joint monitoring and evaluation of client outcomes and the impact of Income Assistance Reform. Where possible, attention will be given to establishing common data collection protocols (e.g., common definitions, improved data collection activities at the individual level, quality assurance standards), exploring the potential of shared systems for the future, limiting administration/reporting burden, and creating a complete understanding of the client experience (i.e., a seamless case management process, addressing current data gaps) as Income Assistance Reform clients reduce their dependency on Income Assistance program through the combined departmental efforts.

Recommendation 3: It is recommended that ESDC communicate to all First Nations Job Fund service providers the data capture protocols that enable the accurate and consistent recording of the client status of being on Income Assistance.

The evaluation identified a data entry error for First Nations Job Fund clients in Alberta and Quebec. ESDC acknowledges that there will always be a risk for data entry errors and, as part of its ongoing monitoring of all Aboriginal labour market programs, has taken measures to reduce the likelihood of these errors, including ongoing monitoring of data quality and follow-up. Nevertheless, errors can happen, especially with new data entry staff or a new program as with the First Nations Job Fund.

By February 15, 2016, by way of a written communication from Service Canada regional directors, ESDC will remind the First Nations Job Fundservice providers of the importance of capturing and sharing complete client records for all First Nations Job Fund participants. ESDC Headquarters will monitor data quality and, if required, work with Service Canada and First Nations Job Fund service providers to address any deficiencies on a timely basis.

Recommendation 4: It is recommended that INAC explore the feasibility of introducing multi-year funding agreements for Enhanced Service Delivery.

INAC does not currently have the program authorities to introduce multi-year funding agreements for Enhanced Service Delivery providers.

The Department acknowledges that annual funding renewal can be a complex process for Enhanced Service Delivery providers. Consideration will be given to introducing multi-year funding agreements in future policy decisions.

2. Action Plan

I recommend this Management Response and Action Plan for approval by the Evaluation, Performance Measurement and Review Committee

Original signed by:

Michel Burrowes
Director, Evaluation, Performance Measurement and Review Branch

I approve the above Management Response and Action Plan

Original signed by:

Paula Isaak
Assistant Deputy Minister, Education and Social Development Programs and Partnerships

1.1 Overview

The evaluation of the On-Reserve Income Assistance Reform initiative was conducted in accordance with the Treasury Board's 2009 Policy on Evaluation. The evaluation examined the Income Assistance Reform's relevance, performance, and design and delivery from its inception in April 2013 to July 2015. Where feasible, the evaluation measured early outcomes to inform the potential funding renewal processes in 2016-2017.

The evaluation was conducted jointly by the Evaluation, Performance Measurement and Review Branch at Indigenous and Northern Affairs Canada (INAC) and the Evaluation Directorate at Employment and Social Development Canada (ESDC).

1.2 Background

On-reserve, Income Assistance is funded by INAC, which allocates resources to First Nations or First Nation organisations, and the province of Ontario under the 1965 Agreement, to deliver Income Assistance on-reserve in the ten provinces. INAC also delivers the Income Assistance program to non-self-governing indigenous residents in the Yukon Territory.

Expenditures for Income Assistance grew from $599 million in 2001-02 to $838 million in 2011-12. This growth (approximately 29 percent) was partially due to growth in the on-reserve population^{Footnote 6} and increases in provincial Income Assistance benefits and rates, to which INAC Income Assistance rates are tied through policy.^{Footnote 7} Please refer to Annex B for a map showing First Nations' participation in the Income Assistance program in 2013.^{Footnote 8} In the late 1990s, provinces and territories began restructuring their respective social assistance programs. A combination of more restrictive measures, enhanced employment and training services, and economic growth led to a reduction in provincial social assistance dependency rates.

While INAC was provided with the authorities for the implementation of active measures in 2003, initially, no additional funding was allocated. Active Measures are activities that help Income Assistance clients increase their employability and find jobs (e.g., basic and life skills, formal education and career counselling, apprenticeships, wage subsidy to employers to hire clients, etc.).^{Footnote 9} Subsequently, beginning in 2009-2010 and for a period of three consecutive fiscal years, INAC internally re-allocated approximately $2 million annually to support active measures pilot projects. In 2012-13, INAC reallocated $14 million to enhance the service delivery capacity and support active measures in 150 First Nations, as a precursor to longer-term active measure supports. Although Income Assistance remains generally a "passive" cheque cutting program, aside from these additional funding, First Nations organisations Income Assistance providers are increasingly implementing active measures using various sources of funding, including provincial funds.

ESDC has a long history of funding and administering the federal government's primary skills development and training programs to help support the participation of Aboriginal peoples in the labour force. It is the lead federal department in the funding of labour market programming, providing an important vehicle for active measures initiatives. The Aboriginal Skills and Employment Training Strategy program, administered by ESDC, invested $1.68 billion between 2010 and 2015, and aims to improve labour market outcomes for all Aboriginal peoples by supporting demand-driven skills development,^{Footnote 10} and fostering partnerships with the private sector and provinces and territories. The Aboriginal Skills and Employment Training Strategy currently supports a network of more than 80 agreement holders^{Footnote 11} with over 600 points of service across Canada.

1.2.2 Program Implementation

The implementation roll-out is incremental and based on a directed proposal process targeting willing and "ready" organizations with more advanced capacity.^{Footnote 15} To date, 27 Enhanced Service Delivery providers (representing 88 First Nations) have been selected to participate in the Enhanced Service Delivery (see Annex C). Potential participating organizations must meet the following five essential criteria:

• Minimum caseload and sustainable over time (50:1 ratio clients to caseworker);
• Excellence in program management;
• Commitment by the Band Council to implement mandatory requirements;
• Labour market potential; and
• Current capacity.

In Ontario, the Income Assistance Reform aims at increasing the number of First Nation organizations that deliver the Employment Assistance component (pre-employment activities) of Ontario Works. The process is initiated by First Nations organizations, which must submit a service plan to the Government of Ontario.

Each Enhanced Service Delivery proposals must be paired between an Enhanced Service Delivery or an Ontario Works provider with a First Nations Job Fund provider for skills development training.

The Income Assistance Reform is structured to enable both Enhanced Service Delivery service providers and FNJF service providers to deliver an enhanced and streamlined case management system. Collaboration between the Enhanced Service Delivery and the FNJF service providers is considered to be a key component of the Income Assistance Reform, from the initial assessment of the client all the way through to the client finding employment.

Enhanced Service Delivery and the FNJF services providers are required to develop a Client Referral Strategy, which establish service standards between both parties and outline how the organizations will work together to serve Income Assistance clients.

Enhanced Service Delivery service provider's role is to identify clients' individual barriers to employment, and establishing an action plan to address these barriers through available programs and services. It is focusing on activity areas such as: pre-employment training on basic literacy skills; life-skills training; essential skills training; high school diploma completion/equivalency; psychosocial interventions; and financial incentives for participation in mandatory provincial or territorial intervention.

FNJF service provider's role is to provide skills development training to secure employment or return to school, as well as other types of active measures such as: skills assessments; personalized training; coaching; and other supports for young Income Assistance clients, on-reserve.

The Enhanced Service Delivery and FNJF Continuum (see Figure 1) illustrate the distinct roles and responsibilities of ESD-providers vis-à-vis FNJF-providers.

 

Text alternative for Figure 1: Enhanced Service Delivery and FNJF Continuum

This diagram demonstrates the movement of clients between Enhanced Service Delivery and FNJF Continuum. The diagram is made up of four main parts. At the top left, a figure of a person represents entry of a client to the Enhanced Service Delivery program as a "ESD Funded Recipient". The client first enters the Intake and the Eligibility and Need Determination phase of Enhanced Service Delivery which contains four sequential steps.

• "Initial Contact"
• "Informal Needs Assessment"
• "File Opened"
• "Determination of eligibility and benefits: Basic Eligibility assessment; and Financial Needs test."

At the "Informal Needs Assessment" step, a client may be referred to other services. At the "Determination of Eligibility and Benefits" stage, a client may either have eligibility denied and exit IA or be referred to other services; or move to "Formal Employability Assessment for mandatory participation," the first of four steps a client can move through in the second phase labelled "Assessment, Planning [and] Implementation."

The second phase of Enhanced Service Delivery contains four steps, each connected to the other by a dual-direction arrow, indicating client movement forwards and backwards between these sequential steps:

• "Formal employability assessment for mandatory participation"
• "Mandatory action plan"
• "Referral"
• "Client Supports: basic pre-employment measures; financial incentives; and, transitional supports"

Client supports may be the final step for ESD clients, or they may move back to the "referral" step.

At the "referral" step, a client may move to the step labelled "Communication/coordination protocols between case managers for client referrals, follow up, and compliance." At this stage, the client may become an "FNJF Funded Recipient" or be referred to other partners' providers "assessment planning interventions." Case managers may also refer clients at this stage back to the "determination of eligibility and benefits" step in the first phase of ESD.

FNJF Funded recipients finally move through four sequential steps:

• "Initial contact"
• "File opened"
• "Assessment and planning"
• "Active Measures: skill development; literacy training and adult basic education; labour market training supports (e.g. transportation and child care)".

"Active Measures" represent the final step for FNJF funded recipients.

2.1 Evaluation Scope and Timing

The evaluation examined the design and delivery of the Income Assistance Reform, including its relevance and early performance from its inception in April 2013 to July 2015. Terms of Reference were reviewed by INAC's Evaluation, Performance Measurement and Review Committee, and were approved on September 9, 2014. ESDC's Departmental Evaluation Committee approved the evaluation strategy in January 2014, and the evaluability assessment on September 30, 2014. The field work was jointly conducted by INAC and ESDC evaluators from February to July 2015.

2.2 Evaluation Methodology

The following section outlines the evaluation's data collection methods, major considerations, strengths and limitations, and quality assurance processes.

2.3 Roles, Responsibilities and Quality Assurance

Due to the joint nature of the evaluation, the following quality assurance activities were put in place:

• The Joint Evaluation Working Group: This group was comprised of departmental program representatives and chaired jointly by INAC and ESDC evaluation managers. The group was responsible for reviewing, validating and commenting on the choice of methodology, the preliminary findings and the final report.
• The Evaluation Advisory Committee: This group, co-chaired by INAC and ESDC evaluation senior officials and consisting of INAC and ESDC evaluation and senior program officials, provided input and direction for the implementation of the evaluation. This advisory committee was responsible for ensuring that all the necessary evaluation issues and questions were covered within the context of available resources, and for the approval of the evaluation report.
• INAC's Evaluation, Performance Measurement and Review Committee: This committee is headed by the Deputy Minister of Indigenous and Northern Affairs. It is responsible for approving the evaluation's Terms of Reference, preliminary findings report, the final evaluation report and Management Response and Action Plan.
• ESDC's Departmental Evaluation Committee: This committee is chaired by the Deputy Minister of ESDC and is responsible for departmental approval of the evaluation strategy, the final evaluation report, and Management Response and Action Plan.

3.1 Continued Need for Income Assistance Reform

Historically, Canada's Aboriginal peoples have experienced higher rates of unemployment and Income Assistance dependency and lower rates of labour force participation than the general Canadian population. Currently, unemployment rates for First Nations people on-reserve are twice the rate of the general Canadian population.^{Footnote 16} Data from 2012-2013 indicate that on-reserve Income Assistance dependency rates were 33.6 percent, compared to approximately five percent among the general Canadian population. This number has remained relatively unchanged over the past decade.^{Footnote 17} The marked increase in Income Assistance support expenditures is largely due to growth in the population on-reserve, and increases in on-reserve Income Assistance rates proportional to provincial rates. Research also shows that there has been a shift towards active measures in North America and Europe. This shift is based on findings that investment in skills development is a key component to reducing income assistance dependency.^{Footnote 18}

The Aboriginal population is not only growing faster than the Canadian population in general, but is significantly younger as well. The 2011 National Household Survey found that 18.2 percent of the total Aboriginal population was between 15 to 24 years of age, in comparison to 12.9 percent of non-Aboriginal population. The high rate of Income Assistance usage among Aboriginal peoples and the proportionately larger youth populations suggests there is potential for increased Income Assistance support in the future.

Aboriginal youth who participated in the Income Assistance Reform also face barriers that inhibit employment readiness and labour market participation. Respondents from communities noted that these impediments are often interconnected and include behavioural, social, and educational and external barriers (e.g., lack of economic development and employment opportunities near their communities, remoteness of communities, etc.,) that require significant time to overcome. Interview participants highlighted the importance of the initial life skills supports provided by the Enhanced Service Delivery to address these issues, noting that they allowed clients to overcome feelings of doubt, low self-esteem and low self-worth.

Figure 2 describes the proportion of clients served by the FNJF from April 2013 to June 2015 who during the development of their Employment Action Plan cited a barrier to employment. At the point of entry into FNJF, over half of clients (56 percent) self-identified as having one or more barriers to employment. The most commonly cited barriers to employment were a lack of work experience (26 percent), lack of transportation (25 percent), and a lack of educational attainment (23 percent). Just under half of clients (44 percent) reported having no barriers to employment.

Source: ESDC’s Aboriginal Standard Data File (April 2013 to June 2015). N=1,272. Note - Clients could provide more than one response.

Text alternative for Figure 2: Barriers to Employment Cited by FNJF Clients

This is a bar graph describing in percentages the barriers to employment cited by FNJF clients. On the y-axis, the types of barrier to employment are listed. On the x-axis, the percentages of clients identifying each barrier to employment are listed. The total percentages reported do not equal to 100% as clients were permitted to report more than one type of barrier to employment.

In descending order, listed from most to least frequently cited, the eleven identified barriers to employment were: 26% identified lack of work experience; 25% identified lack of transportation; 23% identified a lack of education; 18% identified remoteness; 16% identified economic; 15% identified lack of marketable skills; 15% identified lack of labour force attachment; 5% identified dependent care; 5% identified other barrier; 3% identified physical, emotional, or mental health; and 2% identified language.

A lack of economic development and employment opportunities within the local communities were also cited as challenges by interview and site visit respondents, especially in regions heavily dependent on the natural resource sector. In such regions, Aboriginal youth can find themselves in competition with applicants who have greater levels of education and work experience, even for entry-level employment. This issue can be exacerbated further in situations where employers are unwilling to hire Aboriginal applicants due to prejudices. In addition, clients are often reluctant to access employment off-reserve, if it means leaving behind their established family and community supports.

The lack of reliable transportation was also identified by caseworkers as one of the most significant barriers to accessing labour market programming and employment. Approximately 25 percent of FNJF clients identified lack of transportation as a barrier to finding employment (see Figure 2). The transportation limitations most often cited during site visit interviews as impacting clients included: the lack or loss of a driver's license; the lack of a reliable personal vehicle (especially difficult to afford if the only employment opportunities are at minimum wage); the distance between reserves and towns; dangerous highways; and a lack of road access. Depending on the province, a driver's license can take up to three years to obtain. In addition, obtaining employment in some remote communities requires access to multiple modes of transportation (bus, boat, floatplane and/or automobile). A practice, found at one site visit, which may warrant further examination is the arranging of a shuttle service for clients to be able to attend their labour market training.

FNJF clients reported a lack of educational attainment as a barrier to employment (23 percent). The on-reserve Income Assistance data (2013-2014) shows very low educational attainment levels for the majority of Income Assistance clients who are classified as able to work. According to the data, 82 percent of Income Assistance clients did not have a high school diploma, while 15 percent of clients had completed high school, and three percent had some post-secondary education.^{Footnote 19} Low educational attainment may impede participation in education and employment programing and also lead to increased dependency on Income Assistance for these youth. According to caseworkers, motivational and self-esteem issues may also impede their participation. Overcoming these issues takes time (often months or more) and resources in regards to supports and case management.

While Figure 2 shows dependent care (five percent) as one of the least commonly cited barriers to FNJF participation for clients, anecdotal evidence from site visits identified that finding childcare was one of the most important barriers for finding and keeping a job. INAC does not currently collect information on barriers to participation in Income Assistance Reform, specifically entry into the Enhanced Service Delivery. As a result, it is unclear if a lack of childcare is preventing some possible entrants into Income Assistance Reform participation. In 2013-14, approximately 20 percent of Income Assistance clients were single parents, with 17.9 percent being women and 2.1 percent men (See Table 2). Recent studies have shown that participants in Active Labour Market programs who are single parents tend to report that the cost of childcare and its lack of accessibility creates an obstacle to participating in job-searching activities, pre-employment training, and securing full time work.^{Footnote 20}, ^{Footnote 21} When single parents are eligible for Income Assistance and unable to afford upfront childcare costs, there is a disincentive for employment.^{Footnote 22} Findings from a 2009 meta-analysis of active labour market programs found that those interventions which provided childcare support to single parents in addition to pre-employment training were more effective at reducing Income Assistance receipt than those which did not.^{Footnote 23}

The evaluation examined the Community Well-Being Index for the communities selected to participate in the Income Assistance Reform. The Community Well-Being Index (2011) is a method of assessing socio-economic well-being at the community level. It is made up of four components based on information collected on community members: Income; Education; Housing; and Labour-force activity. The score ranges from a low of zero to a high of 100. In 2011, non-Aboriginal communities had an average Community Well-Being Index score of 79, while First Nations overall had a score of 59. When communities participating in the Income Assistance Reform are examined, the average Community Well-Being Index score is 54. However, scores range from a low of 37 to a high of 74, indicating a great deal of variability among communities in terms of socio-economic well-being of its members.

3.2 Alignment with Federal Government Priorities

As part of the 2009 Federal Framework for Aboriginal Economic Development, the federal government seeks to foster linkages between Income Assistance, education and labour market programing to support improved participation of Aboriginal peoples in the economy.^{Footnote 24} In an effort to accomplish this goal, Income Assistance on-reserve was reformed, moving from passive delivery of income supports to "active" labour interventions.

The Enhanced Service Delivery model, as part of the Income Assistance Reform, is aligned with INAC's mandate to "support Aboriginal peoples and Northerners in their efforts to: improve social well-being and economic prosperity; develop healthier, more self-sufficient communities; and participate more fully in Canada's political, social and economic development – to the benefit of all Canadians."^{Footnote 25} The Income Assistance Reform falls under INAC's strategic outcome area, "The People", which is intended to promote "individual, family and community well-being for First Nations and Inuit."^{Footnote 26}

The Income Assistance Reform is also aligned with ESDC's mission statement to "build a stronger and more competitive Canada, to support Canadians in making choices that help them live productive and rewarding lives and to improve Canadians' quality of life."^{Footnote 27} The 2015-2016 Departmental Report on Plans and Priorities supports FNJF's alignment with ESDC's priorities: particularly the departmental priority of maximizing the labour market participation of under-represented groups. FNJF is also consistent with ESDC's strategic outcome of producing a "skilled, adaptable and inclusive labour force and an efficient labour market."^{Footnote 28}

While INAC has had authority to implement active measures since 2003, only strategies geared towards enforcing mandatory skills training or work requirements for employable candidates on-reserve were implemented as small pilot projects in select communities.^{Footnote 29} Budget 2012, Canada's Economic Action Plan, acknowledged that Canada's Aboriginal population is the youngest and fastest growing segment of the general population. As such, the Budget aimed to invest in "equipping First Nations People with the skills and opportunities they need to fully participate in the economy."^{Footnote 30} Budget 2013 reaffirmed this goal with increased funding equalling $241 million over five years to improve the on-reserve Income Assistance Program.^{Footnote 31}

Through the introduction of active measures via the Enhanced Service Delivery, FNJF, and related programs, INAC and ESDC seek to keep pace with the provinces and Yukon in reducing Income Assistance dependency on-reserve.

3.3 Alignment of Roles and Responsibilities

The Income Assistance Reform is funded by the federal government and administered by First Nations or First Nations organizations in seven provinces and in the Yukon Territory.^{Footnote 32} The funding relationship of the federal government relative to health, welfare and social spending is affirmed in the Constitution Act Section 91(24).^{Footnote 33} However, First Nations are able to administer bylaws and deliver social services where sufficient administrative capacity exists.^{Footnote 34}

Assuming they have sufficient capacity for labour market support for Income Assistance clients, Income Assistance service or Aboriginal Skills and Employment Training Strategy delivery agents are likely the most appropriate entities to implement active measures programming and make referrals to FNJF. Some interviewees mentioned that provincial governments were in a better position to implement both Income Assistance and active measures, but this would require provinces to enter into agreements with INAC respecting service delivery on-reserve, and the benefit of this approach over the current approach was not made apparent.

With respect to the delivery of the Income Assistance Reform at the community level, key informant and site visit respondents reported that it was important to ensure funding recipients maintained autonomy over how funding was delivered. Funded communities and agreement holders were viewed as being in the best position to determine how best to direct supports and services to address the needs of clients.

4.1 Program Governance

The Income Assistance Reform governance structure is designed to support a joint approach to the delivery of the Initiative. It is comprised of national and regional representatives from INAC and ESDC. Three governance structures were developed to support the delivery of the Income Assistance Reform:

• An Oversight Committee composed of senior ESDC and INAC officials from National Headquarters and the regions, as well as representatives from Health Canada and Treasury Board Secretariat. This Committee provides strategic direction and monitors the overall implementation of Income Assistance Reform, including the allocation of resources nationally.
• A Program Review and Advisory Committee consists of reviewing the Enhanced Service Delivery and FNJF funding proposals recommended by the Regional Implementation Teams to ensure consistency of implementation across the country, adherence to the reform objectives and efficient allocation of the budget nationally. It also reviews regional strategic plans for national and inter-departmental coherence, identifies policy or programmatic issues requiring consideration by the Oversight Committee, and provides advice and direction to regions.
• A Regional Implementation Team was created in each region and is composed of regional staff from INAC and ESDC as well as one or more representative(s) from INAC Headquarters. The primary role of each Regional Implementation Team is to develop Joint Regional Implementation Plans. There, they identify, develop, assess and recommend individual Enhanced Service Delivery and FNJF active measures projects for consideration by the Program Review Advisory Committee, and follow-up on their progress. The teams work directly with Enhanced Service Delivery providers and FNJF Aboriginal Skills and Employment Training Strategy agreement holders to ensure that all reform criteria and objectives are met in the development of proposals. The teams also ensure that funding approval processes at ESDC and INAC are coordinated to ensure timely funding decisions and implementation of the Enhanced Service Delivery and FNJF programming.

In late 2014-2015, changes were adopted to all three governance structures in an effort to streamline efforts, avoid duplication, and improve communications. Respondents reported that this resulted in committees of a more manageable size and improved efficiencies in terms of accountability and decision making. Some regional directors interviewed noted that they were not informed of the changes to the make-up of the Program Review and Advisory Committee, and as a result, reported not always being aware of information exchanges.

Overall, evidence from the document review and key informant interviews suggest that the mandate, roles and responsibilities of the Oversight Committee, the Program Review Advisory Committee, and the Regional Implementation Teams are distinct from each other and that there is no duplication between these governance bodies. Roles and responsibilities are clearly laid out in program documents and modifications have been made to the governance structure based on feedback from committee members. It appears that there is regular communication between the different levels.

Joint Regional Implementation Plans were produced by the Regional Implementation Teams. According to program officials interviewed, the Joint Regional Implementation Plans development process had to be completed in a short period of time. Some program officials interviewed found it challenging to find relevant Labour Market Information, including accurate estimates of the size of the target cohort population.

Finally, some program officials pointed out that approval processes between departments tended to slow down the implementation of the program. For example, departments were rarely, if ever, synchronized with one another with respect to their own decision-making processes. As a result, this lack of synchronicity at times created delays in decision making.

While the Joint Regional Implementation Plans for each region outline the roles, responsibilities, communication processes, tools and guidelines at the community/site level, site visit respondents reported a lack of clarity in this regard. In particular, respondents reported some disagreement between the Enhanced Service Delivery and FNJF providers in terms of the types of supports and services eligible for funding under each component, as well as different perceptions as to when a client is "employment-ready." In some communities, this disagreement has led to a breakdown in communication, as well as trust issues between Enhanced Service Delivery and FNJF providers. To address these issues, site visit respondents reported that additional support and guidance was required from the federal level in order to clarify the roles and responsibilities of Enhanced Service Delivery and FNJF providers, as well as to mitigate conflicts.

4.2 Roll-Out of Income Assistance Reform in First Nation Communities

The document review, key informant interviews and site visit participants identified a number of factors that affected the roll-out of the Income Assistance Reform in participating communities.

First, it took longer than anticipated for the Enhanced Service Delivery to be launched. This meant that in Year 1 of funding (2013-14), the Enhanced Service Delivery was only available for three months of potential program delivery and funding. In addition, some Enhanced Service Delivery funding recipients had to build their capacity to deliver the Enhance Service Delivery, including the hiring and training of staff. INAC program officials stated that these delays resulted in the Enhanced Service Delivery not being fully functional until April 2014. Despite delays, caseworkers expressed pride in their ability to implement the Enhanced Service Delivery. Interviewees stated that their previous experience in implementing case management activities facilitated making programs operational in a short time period, thus, validating the approach of targeting Income Assistance providers who were identified as having advanced capacity for early implementation.

Secondly, while no percentage was provided, according to a few program staff interviewees, some Enhanced Service Delivery funding recipients demonstrated less capacity than originally thought with respect to their data tracking and delivery infrastructure. In addition, some site visit respondents reporting that they expected more support from INAC to build this capacity. Interviews with community stakeholders, caseworkers and program officials revealed that limited guidance and materials, such as templates to collect necessary data, were provided to Enhanced Service Delivery funding recipients. Some documents were provided in draft form, which funding recipients were reluctant to use. This left caseworkers to develop their own tools to support the delivery of Enhanced Service Delivery. The revised data collection instrument that includes fields for Enhanced Service Delivery activities has since been revised and should contribute to addressing this issue.

4.3 Program Design

The introduction of on-reserve Income Assistance Reform moves Income Assistance delivery away from a passive "cheque cutting" exercise to a more client focused assistance program, by introducing active measures and case management practices. All interviewees agreed that the introduction of case management benefits clients, and assists them to move towards employability, and ultimately employment. During the course of interviews, the following areas were identified for potential adjustment to program design.

4.4 Provincial Service Delivery: The Ontario Works Example

In Ontario, the equivalent of the Enhanced Service Delivery is delivered by Ontario Works. As of April 1, 2013, out of 110 eligible communities, 63 First Nations have access to the Employment Assistance Services of Ontario Works. Additional funding under Income Assistance Reform was provided to Ontario to increase the number of First Nations communities into Ontario Works. As of July 2015, 71 First Nations had access to the Employment Assistance Service of Ontario Works.

As part of Ontario Works, administrators are responsible for case managing clients and require that all applicants for financial assistance must enter a Participation Agreement. The Participation Agreement, similar to an action plan under the Enhanced Service Delivery model, outlines a client's immediate and longer-term goals, and the activities they will undertake to improve their employability while on Income Assistance. In some cases, a Participation Agreement can identify that the individual does not need to actively participate, such as a sole support parent or a caregiver of a disabled family member. Ontario Works Administrators have the discretion to impose measures for recipient non-compliance with the participation expectations set out in their Participation Agreement. Persons with disabilities in receipt of the Ontario Disability Support Program are given the opportunity to voluntarily participate in Ontario Works employment assistance activities leading to employment. However, disabled applicants who are voluntarily participating in active measures are not subject to consequences for noncompliance.

In large measure, the federal implementation of Income Assistance Reform on-reserve in Ontario is a matter of providing FNJF supports, given Ontario Works already provides case management and active measures. Thus, case study participants did not raise the same implementation issues highlighted by other service delivery agencies discussed in this report, largely because to some degree, the capacity has been in place and operational since 1965. The roles and responsibilities between Ontario Works caseworkers and FNJF were clear and understood. No major issues were raised with respect to the implementation of the FNJF component beyond those already raised in this report.

As discussed in Section 3.3, provincial governments, from the standpoint of embedded experience and policy, have a strong foothold in Income Assistance and in delivering active measures programs. The Ontario model may be working largely due to historical capacity development on-reserve for service providers over the past 50 years. The success of the model does not suggest that it would be an appropriate approach in other provinces, as capacity would still have to be developed in much the same way as it does now. That said, in Ontario at a minimum, the approach appears to be optimal.

5.1 Data Tracking and Performance Measurement

5.2 Performance of Income Assistance Reform

Income Assistance Exits

During 2013-2014 and 2014-2015, 2,041 clients exited from Income Assistance, exceeding the target of 1,388 clients for that period by 653 clients (See Table 5). By March 2017, Income Assistance Reform is expecting 4,277 clients to exit Income Assistance.

Table 5: Income Assistance Client Exits from Income Assistance, Aged 18 to 24 (2013-2014 to 2014-2015)

Fiscal Year	2013-2014	2014-2015				Total
Fiscal Quarter	Q4	Q1	Q2	Q3	Q4
Client Exits from Income Assistance to Employment or Education (per quarter)	108	376	497	490	570	2,041
Source: INAC's Income Assistance / Enhanced Service Delivery Data Collection Instrument.

 

An analysis of Income Assistance exits for 18 to 24 year olds in the 27 participating communities in the year prior to (2013-2014) and the year after the Enhanced Service Delivery introduction (2014-2015) was conducted (See Figure 3). It found that in 2013-2014 approximately seven percent of 18 to 24 year olds exited from Income Assistance. The following year, this increased by 22 percentage points to 29 percent, suggesting that Income Assistance Reform had an effect on the number of individuals exiting from Income Assistance to either employment or education.^{Footnote 42}

Source: INAC's Income Assistance / Enhanced Service Delivery Data Collection Instrument.

Text alternative for Figure 3: Change in the Mean Proportion of Exits Relative to the Total Number if Income Assistance Clients, Aged 18 to 24 (2013-14 to 2014-15)

This is a bar graph which illustrates the changes in the mean proportion of exists relative to the Total Number of IA Clients, aged 18 to 24, between the fiscal years 2013-2014 to 2014-2015. The x axis illustrates the two fiscal years examined, 2013-2014 and 2014-2015. The y axis lists percentages from 0% to 35%. There are two bars. The first bar is for fiscal year 2013-2014, and it show the number of exits from IA as 7%. The second bar, representing fiscal year 2014-2015, shows the number of IA exists has increased to 29%.

 

Interviewees stated that clients who participated in and completed the Enhanced Service Delivery programming were more employable due to the assets or skills acquired during their training. Interviewees also stated that the benefits of participation in the Enhanced Service Delivery were greater than just increased employability; clients also showed higher levels of motivation and increased self-esteem.

5.3 Efficiency and Economy

It is too early in the Initiative's lifecycle to reliably assess cost-effectiveness. Future evaluation work will be better positioned to examine this as well as the incremental impacts of Income Assistance Reform.

The evaluation found that the proportion of total spending on all aspects of Enhanced Service Delivery, including internal operations, maintenance, and salary comprises less than one percent of program expenditures. This suggests a strong level of program efficiency from the point of view of internal spending versus grants and contributions (Refer to Annex E for further details).

A simple cost analysis, based on ESDC contributions, was undertaken to calculate the average FNJF participant cost for 2014-2015. The Aboriginal Skills and Employment Training Strategy evaluation findings served as the most appropriate comparison for FNJF. In 2014-2015, FNJF service providers claimed an estimated $6.2 million in expenses, serving 1,093 clients. The average cost per client for FNJF is estimated at $5,650 for 2014-15. This is lower than the Aboriginal Skills and Employment Training Strategy participants at $6,100 in 2012-13.

The evaluation revealed some areas of potential inefficiencies:

Funding for Enhanced Service Delivery is proposal based and approved annually. FNJF service providers, on the other hand, enter into multi-year funding agreements. Interviewees identified the need to have secure multi-year or ongoing funding for the Enhanced Service Delivery, in order to allow communities to properly plan for the future. Waiting to be notified about funding renewals was identified as creating "stops and starts" with programming, resulting in uncertainty and stress for clients and staff. All site visit interviews revealed that the late notification of 2014-2015 funding resulted in programming slowdowns and uncertainty. Three sites had begun to wind down activities, lay off staff, and paused client interventions while awaiting funding approval. A cascading effect was felt by FNJF, as slowdowns in Enhanced Service Delivery influences when clients can be referred to FNJF, due to paused Enhanced Service Delivery client interventions.

Recommendation 4: It is recommended that INAC explore the feasibility of introducing multi-year funding agreements for the Enhanced Service Delivery.

5.4 Potential Integration between the Enhanced Service Delivery and FNJF

Considering the communities included in the evaluation, the level of integration between the Enhanced Service Delivery and FNJF varied by community. In some of the studied communities, the Enhanced Service Delivery and FNJF are more fully integrated. For example, in Quebec, a whole team worked closely together to assess, assign interventions and monitor client progress. At one of the sites visited in British Columbia, evaluators observed one caseworker being responsible for the delivery of both funding streams. The caseworker continually assesses clients and moves them from one intervention to the next, often without the clients even being aware that they are benefiting from different funding sources. In other communities, there are distinct Enhanced Service Delivery and FNJF caseworkers with separate offices and case files per client. For this latter design, the process may not be as seamless and under this approach, a few issues were raised about difficulties when referring clients from the Enhanced Service Delivery to FNJF.

In a study conducted by Brock and Harknett (1998) that randomly assigned individuals to either an integrated or a dual caseworker model, the authors found that an integrated case management approach resulted in a significant difference in:

• The rate of referral for sanctions;
• The rates at which clients were able to maintain at least 15 hours/week of employment; and
• Rates of participation in workshops, job searches, and training activities.

In all the cases where the difference was significant, integrated cases produced significantly better employment rates, earnings and reduced Income Assistance reliance for participants relative to the comparison group.^{Footnote 44}

Building an ongoing relationship with clients is one of the most important elements of the case management process, as it establishes a level of trust required to move a client forward through their Action Plan. Key informants indicated that the transfer from Enhanced Service Delivery to FNJF caseworkers may interrupt client progression towards employability, as the client needs to re-establish a trusting relationship with their new caseworker.

The distances between the Enhanced Service Delivery and FNJF offices were also identified as a barrier for clients to access FNJF. In several locations visited by evaluators, the FNJF office was over 30 kilometers outside of the community, and in one situation, the FNJF office was over 200 kilometers away. To compensate, some Enhanced Service Delivery caseworkers are taking on additional FNJF type tasks (e.g., speaking with employers on behalf of clients, buying equipment for clients' jobs, etc.). As was previously mentioned, many clients face transportation issues that prohibit them from travelling far distances to access programming. Consideration should be given to improve client access to Enhanced Service Delivery and FNJF services. This may be achieved by offering combined services through the service provider who is in closest proximity to clients in situations where distance between offices prevents easy access by clients.

Based on key informant interviews and site visits, there is potential for overlap and duplication with respect to the roles and responsibilities of Enhanced Service Delivery and FNJF service providers. There is evidence that prior to Income Assistance Reform, both Income Assistance and the Aboriginal Skills and Employment Training Strategy agreement holders offered life and essential skills supports, as well as employment training. With the creation of Income Assistance Reform, the expectation was that these interventions were to be delivered separately and in a complementary manner by each proponent. After the implementation of Income Assistance Reform, some duplication in program activities was still observed since work continued to be carried as it had previously been done in the past by each Enhanced Service Delivery and FNJF service provider. Furthermore, with two funded service delivery organizations and a two caseworker system, clients are case managed by two caseworkers; both caseworkers create action plans and refer clients to relevant interventions (pre-employment supports vs job training); and client information is tracked in two separate databases.

One possible consideration for future programming is the simplification of the funding application and reporting processes for service providers. For example, the current application process requires that First Nations service providers complete two funding applications, one for INAC's Enhanced Service Delivery and another for ESDC's FNJF. The process could be simplified by having First Nations submit a single funding application to one department that addresses both Enhanced Service Delivery and FNJF funding streams. Similarly, the current reporting process requires First Nations to prepare two reports, one delivered to INAC for Enhanced Service Delivery and another delivered to ESDC for the FNJF. Simplifying the proposal and reporting processes would reduce the administrative burden on First Nation communities. If INAC was able to move to multi-year funding agreements, as recommended in this report, the above consideration would be possible.

Additionally, efforts should be made to ensure clients have easy access to personalized services by integrating the provision of both Enhanced Service Delivery and FNJF services. This is particularly important in rural communities where distance may be a barrier to accessing program services. An integrated, seamless case management process would eliminate the need for a client referral strategy; eliminate ambiguities with respect to roles and responsibilities inherent in a two caseworker approach; require a single client assessment and action plan; and track all interventions by client resulting in an improved analysis and understanding of clients for monitoring, program and policy decisions, and evaluation purposes.

6.1 Conclusions

The Government of Canada needs to ensure the modernization of Income Assistance programming on-reserve, and the Income Assistance Reform initiative is a positive step toward implementing active measures^{Footnote 45}. There is a need for programming designed to increase employability of Income Assistance clients, particularly given the high rate of Income Assistance dependency (particularly among Aboriginal youth on-reserve) and increases in Income Assistance expenditures. INAC and ESDC, along with First Nation service, have a significant role to play in this regard. As the funding of social services is within the purview of INAC, it is essential that Income Assistance service providers be provided with the ability to case manage their clients to better promote employability and reduced reliance on Income Assistance. ESDC's role in providing labour market programs falls under the mandate of the Department of Human Resources and Skills Development Act (S.C. 2005, C.34).

The delayed roll-out of the Income Assistance Reform initiative resulted in very limited data to inform performance of the program to date. Additionally, there are some key limitations in the data collection and reporting for both INAC and, to a lesser degree, ESDC. That said, there are strong indications that implementation has led to positive outcomes. For example, there was an observed increase in Income Assistance exits (22 percent) compared to the year prior to program implementation and 2,041 clients exited from Income Assistance Reform to employment or education. It is important to consider the applicability of current policies and guidelines regarding eligibility and reach given the issues faced by First Nation youth. To that end, this evaluation has highlighted several areas where further discussion is needed, including the duration for participation, the target age cohort and process for interruptions in service.

To date, more than 6,800 Income Assistance clients, aged 18 to 24, have entered into Enhanced Service Delivery case management, which exceeds the initial targets. However, initial referrals to FNJF were lower than initially targeted. Several reasons were identified for the lower than anticipated referrals. For example, more clients than expected required greater than the six months of life and essential skills interventions originally forecasted; and direct referrals of job ready clients to employment or to other skills development and job training services. It was expected that referrals to FNJF would increase now that sufficient time has passed to allow for clients to receive required pre-employment skills development supports. Additionally, there are indications that the variety of supports and services received by clients has been positive both for them and employers.

Given that Income Assistance Reform has only been implemented in select communities, it is too early in the Initiative's life cycle to reliably assess the cost-effectiveness of program delivery. A simple cost analysis found that the average cost for an FNJF participant was $5,650 for 2014-15. The evaluation identified some areas of potential efficiency. These include integrating and streamlining the delivery of interventions and examining the feasibility of multi-year funding agreements for Enhanced Service Delivery funding recipients. For example, late notification of 2014-2015 funding led to staff layoffs and paused client interventions in some delivery sites. This inevitably caused delayed referrals to the FNJF.

6.2 Recommendations

Most of the recommendations in this report stem from aspects of data collection that are intended to improve ongoing performance measurement and to support future policy, research, and evaluation work. There were concerns over the completeness and reliability of some of the data. For example, the socio-demographic and intervention data collected by the Enhanced Service Delivery was insufficient to adequately describe participants and the types of interventions in which they participated. An examination of the data collection strategy by both departments is needed to more efficiently capture and link participant data to ensure effective client monitoring and analysis of outcomes and impacts. Finally, funding the Enhanced Service Delivery on a multi-year basis would reduce service interruptions and improve delivery of the Income Assistance Reform. Based on the findings of this evaluation, the following recommendations are made:

Recommendation 1: It is recommended that INAC ensure that data collection instruments and processes used by service providers consistently and accurately capture adequate socio-demographic and program outcome information on clients.

Recommendation 2: It is recommended that INAC and ESDC collaborate to explore ways to more efficiently capture and link information collected by each department to allow for strengthened client monitoring and analysis of outcomes and impacts of the Income Assistance Reform.

Recommendation 3: It is recommended that ESDC communicate to all FNJF service providers the data capture protocols that enable the accurate and consistent recording of the client status of being on Income Assistance.

Recommendation 4: It is recommended that INAC explore the feasibility of introducing multi-year funding agreements for the Enhanced Service Delivery.