Archived - Formative Evaluation of the Department of Indian and Northern Affairs Canada - Special Education Program

Author:Evaluation, Performance Measurement and Review Branch Audit and Evaluation Sector Indian and Northern Affairs Canada
Date: (December, 2007)

PDF Version   (672 Kb, 46 Pages)

Table of Contents

• Part I Introduction, Program Description and Evaluation Methodology
  • 1 INTRODUCTION
  • 2 PROGRAM DESCRIPTION
    • 2.1 Background
    • 2.2 Objective and Expected Outcomes
    • 2.3 Budget
    • 2.4 High-Cost Special Education Needs
    • 2.5 First Nations Regional Managing Organizations
    • 2.6 Special Allotment
    • 2.7 Direct and Indirect Services
    • 2.8 Individual Education Plans
    • 2.9 High-Cost Special Education (HCSE) student identification and assessment
  • 3 EVALUATION METHODOLOGY
    • 3.1 Research approach
    • 3.2 Methodology
    • 3.3 Research Limitations
• Part II Evaluation Findings
  • 4 EVALUATION FINDINGS
    • 4.1 Needs and Contributing Factors
    • 4.2 Early results
    • 4.2.1 High-cost special needs student identification
    • 4.2.2 Province of B.C. data on First Nations special education students
    • 4.2.3 Student Assessments and Individual Education Plans
    • 4.2.4 SEP Expenditures
    • 4.2.5 Full Service and Partial FNRMOs
    • 4.2.6 Culturally sensitive services
    • 4.2.7 Teacher training
    • 4.2.8 Parental Involvement and Community Outreach
    • 4.2.9 HCSE student learning outcomes
  • 5 PERFORMANCE MONITORING
    • 5.1 Nominal Roll
    • 5.2 Annual Reports
  • 6 SHOULD SEP STAY A SPECIAL ALLOTMENT?
• Part III Conclusions, Recommendations and Management Response/Action Plan
  • 7 CONCLUSIONS
  • 8 RECOMMENDATIONS
  • 9 MANAGEMENT RESPONSE / ACTION PLAN
• Appendix A List of Acronyms

Part I: Introduction, Program Description and Evaluation Methodology

1 Introduction

This report has been prepared to fulfill a departmental commitment to complete a formative evaluation of the Special Education Program (SEP), to inform the program's renewal in March 2008, and to provide a foundation for a future comprehensive evaluation of the program.

Consistent with Treasury Board Secretariat expectations of formative evaluations, researchers have looked at the adequacy of the program's design and delivery and progress towards expected outcomes. Because the program was only introduced in late 2002-03, they did not attempt to measure long-term outcomes.

The specific questions were as follows:

• What can be learned about the current and projected level of need and costs for high cost special education for students with moderate to severe disabilities living on reserve? What is known about the factors that contribute to or could reduce child and adolescent disability rates on reserve?
• What expected and unexpected results has the program achieved to date? That is, what has been achieved in terms of meeting student needs, delivering culturally sensitive programs and services which meet provincial standards, and improving learning outcomes and well-being?
• What outcomes are or should the Department of Indian and Northern Affairs Canada (INAC) be tracking, measuring and reporting? That is, are current expectations relevant? Is the right information being gathered to permit the measurement of outcomes? How effective are current reporting procedures?
• What is needed to improve INAC programming for students living on reserve with moderate to severe disabilities? That is, is the current funding and program delivery approach appropriate? Should SEP remain a special allocation or be rolled into the overall education budget? What modifications to the program are required?

Evaluation research covered the planning period prior to the program's introduction in late 2002-03 until early 2007-08, and focused mainly on delivery by Band-operated schools. However, findings about outcome information that should be monitored, measured, and reported on applies to all schools that are funded by the program.

The report has three sections:

Part I, with this introduction, a brief description of the Program and the Treasury Board conditions governing its operation, and an overview of the evaluation;

Part II, presenting evaluation findings; and

Part III, with evaluation conclusions, recommendations, and the Department's Management Response/Action Plan.

2 Program Description

2.1 Background

Although INAC provided some support for high-cost special education services prior to SEP's introduction in 2002-03, the nature and level of this support varied from region to region because there was there no specific program or funding allocation.

In recognition of a serious and growing gap in special needs services for First Nations students living on reserve, the federal government committed in the 2001 Speech from the Throne to working with First Nations to measure and reduce the incidence of fetal alcohol syndrome in the Aboriginal population, and to helping Aboriginal children with special needs. In the 2001 Budget, an additional $60 million over two years was committed to support children with special needs who live on reserve.

In June 2002, the Department received approval to establish a Special Education Program that would cover the costs of special education services for First Nations students living on reserve. The additional $95 million in incremental funding brought the department's special education budget to $248.1 million for three years from 2002-03 to 2004-05.

The Department's goal was to ensure the availability of critical services and support for First Nations students living on reserve who are affected by moderate, severe or profound behavioural and/or physical challenges. As primary and secondary special education programming had become a fundamental component of provinces' and territories' elementary and secondary education systems, it was critical that INAC provide a comparable level of programming for First Nations students living on reserve.

In January 2003, a new Special Education Program and associated Terms and Conditions were approved until June 2005 and, subsequently, to March 2006. In December 2004, the Department sought and received additional special education funding ($10 million for 2005-06, $20 million for 2006-07 and $30 million for 2007-08) to address unmet needs and ensure equitable funding levels to First Nations across Canada.

It was decided at that time that the proposed approval of the Special Education Program renewal would be delayed from March 2006 to March 2008, concurrent with the renewal of other education programs for First Nations living on reserve (Elementary/Secondary Education Program and others) in order to ensure coordination and maximize effectiveness.

2.2 Objective and Expected Outcomes

The objective established for the Special Education Program was to "improve the educational achievement levels of First Nation students on reserve by providing for access to special education programs and services that are culturally sensitive and meet the provincial standards in the locality of the First Nation." [Note 1]

The expected long-term outcome was to allow First Nations special needs students to achieve their fullest potential and be contributing members of society, as well as increase the numbers of high-cost special needs students acquiring a regular high school diploma.

The program was created for all (high-cost special education) First Nation students living on reserve across Canada, except those in Nunavut, Northwest Territories and the Yukon, and in communities that fall under the jurisdiction of the James Bay and Northern Quebec Agreement and the Northeastern Quebec Agreement (for the Naskapis).

2.3 Budget

According to Departmental figures, the SEP budget, including incremental funding, was $51.9 million in 2002-03, $95.1 million in 2003-04, and $101 million in 2004-05 for a total of $248.1 million before the program was extended in 2005. The budget has since grown, rising to $107M in 2005-06 and to $118.1M in 2006-07.

2.4 High-Cost Special Education Needs

Broadly speaking, special education needs of students fall within a continuum of mild to moderate, moderate to severe, and severe to profound needs.

Services for special education students whose needs are mild to moderate (i.e., low-cost special education) are expected to be addressed by INAC's Elementary/Secondary Education Program. Only services for special education students whose needs are more severe (generally ranging from moderate to profound), and whose special education needs cannot be met within the resources identified for the general student population (i.e., high-cost special education), can be funded by the Special Education Program.

According to Departmental planning documents, while provincial definitions vary, high cost special education students are generally students who:

• Are physically dependent
• Have a hearing impairment
• Have moderate to severe behavioural disorders
• Have chronic health impairments or physical disabilities
• Are deaf and blind
• Are autistic
• Have communication disorders
• Are physically dependent

SEP funding is for services for special education students who are on the Nominal Roll, an INAC registry of First Nations students who live on reserve. [Note 2] It is not for enhanced programs and services for gifted students or enriched subject-specific programming for students streamed into, for example, drama or the arts.

The Department arranges for the administration of SEP funding with Chiefs and Councils or their organizations (including organizations operating under Self-Government agreements) or by entering into agreements for service delivery with provincial governments and/or agencies, or private education facilities for the provision of special education services or programs.

Contributions for high-cost special education services (direct and indirect) [Note 3] can be made to Chiefs and Councils or other organizations that they designate (band/settlements, tribal councils, education organizations, political/treaty organizations, public or private organizations engaged by or on behalf of Indian bands), to provincial ministries of education, provincial school boards/districts, and private educational institutions. SEP funding is also used for direct delivery of services by INAC in seven federal schools.

Contributions for the delivery of indirect services can also be provided to First Nation Regional Managing Organizations (FNRMOs).

Resources or services generally provided for high-cost special education students are some combination of:

• additional staff, including, for example, reading assistance teachers, special education teachers, teacher assistants, personal attendants, or teachers of the deaf
• specialist services, including speech / language pathologists, occupational and physiotherapists
• specialized programs assistive technology modifications to physical environments, and/or professional development for school staff who work with these students.
• assistive technology
• modifications to physical environments, and/or
• specialized programs

2.5 First Nations Regional Managing Organizations

At present, there are 18 First Nations Regional Managing Organizations (FNRMOs): eight in Saskatchewan, two in the Atlantic region, four in Alberta, two in Quebec and one each in Manitoba and British Columbia. Ontario does not have any FNRMOs.

The role of FNRMOs is to provide services and support, and in some cases community education, for schools, educators, parents, families and First Nations communities.

Two objectives of the Department's financial support to FNRMOs were to achieve economies of scale and to ensure that individual schools, especially in more isolated and rural areas, would have access to school board-like services that they would have difficulty accessing independently.

Four FNRMOs, with agreement from the First Nations they serve and approval from INAC, coordinate the program's delivery in their catchment area. These "full-service FNRMOs" have authority to allocate SEP funding among schools [Note 4], provide support to First Nation schools and teachers, by way of direct and indirect programs and services, and report to INAC on the full SEP funding used in their catchment area. The other 14 organizations, known as "partial FNRMOs," are expected to provide indirect services in support of schools, but not direct services. They are also not responsible for budget allocations. [Note 5]

When SEP was launched, in late 2002-03, there were two full-service FNRMOs: the First Nation Schools Association/First Nations Education Steering Committee (FNSA/FNESC) in British Columbia and the First Nations Education Council (FNEC) in Quebec.

It was expected that other FNRMOs would gradually transition from partial- to full- service by 2008. Six partial FNRMOs are identified in the 2006 National Program Guidelines as having potential to become full-service FNRMOs: L'Institut culturel et éducatif des Montagnais (ICEM) in Quebec, the Prince Albert Grand Council (PAGC) and Battlefords Tribal Council (BTC) in Saskatchewan, the Manitoba First Nations Education Resource Centre (MFNERC), the Mi'kmaq Kina'matnewey (MK) in Nova Scotia, and the New Brunswick Education Initiative Incorporated (NBFNEII).

2.6 Special Allotment

SEP funding was provided to the Department as a "special purpose allotment," that is, an amount provided for a specific use. Because resources were provided as a special purpose allotment, funding can only be provided by contribution agreement, not by Alternative Funding Arrangements or Flexible Transfer Payments. Funds not spent in any fiscal year must be returned to the government's Consolidated Revenue Fund, that is, funds cannot be transferred to other programs. As well, the program's budget allocation is fixed, that is, funds cannot be added from other programs.

2.7 Direct and Indirect Services

As in provinces, the INAC special education program can cover both direct services to students and indirect services, provided that at least 75% of the budget goes to direct services. As stated in the 2005 Terms and Conditions and shown below, as currently defined, '. these are broad categories [and] it is worth noting that several of the services identified can be delivered at both the school (direct) and regional or RMO level (indirect) :

Table 2.1 Direct and Indirect Services as per SEP Terms and Conditions (2005)

2.8 Individual Education Plans

When a student has been identified as needing high-cost special education, an Individual Education Plan (also called an Individual Learning Plan or Individual Program Plan) is developed. This Plan is the HCSE student's education roadmap: it sets out the student's learning needs, types of interventions or support needed (e.g., special instruction, personnel or materials), and the goals and objectives established for the student (i.e., academic, social/emotional, behavioural, and life skills). [Note 6]

Ideally, the Individual Education Plan is jointly developed by specialized professionals, teachers, parents and, depending on their age, the students themselves.

2.9 High-Cost Special Education (HCSE) student identification and assessment

The process to determine which students should be provided with HCSE services involves both teachers and specialists.

Teachers with appropriate training do preliminary assessments to determine the need for a more in-depth and professional analysis and diagnosis. For students they believe need high-cost special education services, they prepare referral reports. Then a formal assessment of students referred is conducted by a professional, including, for example, an education psychologist, speech or language specialist, and/or physician.

The product of the formal assessment is a diagnosis of the condition(s) affecting the student's learning and treatment recommendations, including special education interventions. This information forms the basis for students' Individual Education Plans.

School personnel then determine the costs of implementing the Plan and develop a detailed budget.

At the time SEP was introduced, the general practice was for schools to wait to start delivering high-cost special education services until formal assessments had been done (the 'assessment-based' approach).

With the program's extension in 2005, the program has increasingly moved towards an 'intervention-based' model of delivery. This approach does not necessarily require formal assessments before intervention strategies are introduced. Teachers with appropriate training are able to use and interpret assessment instruments and, in turn develop the necessary intervention measures to address immediate needs while awaiting more formal assessments.

3 Evaluation Methodology

3.1 Research approach

Research was based mainly on available data and information, an approach shaped by:

• A request from program management to provide information to support planning for renewal of Elementary/Secondary (K-12) programming and advance the evaluation's completion date from late 2007 to earlier in the year;
• An evaluation plan, prepared in 2006, which was developed to meet Treasury Board's requirement for an evaluation framework; and
• Challenges in securing input from First Nation organizations and educators.

3.2 Methodology

Evaluation fieldwork was conducted largely between mid-January and May 2007. Research was done in part by consultants and in part by Evaluation Branch staff; report- writing was by Evaluation Branch staff.

The research included the following data collection activities:

Examination and review of Documents, Data and Research:

• SEP documents including governing documents, National Program Guidelines,
• First Nation, INAC and provincial educational policy, programming and evaluation reports; [Note 7]

• Education studies commissioned by INAC and First Nations; Nominal Roll and Annual Report data; and Other documents, data and research, for example, federal disability statistics, studies and reports of the Auditor General and Treasury Board, and recent research on special education and disabilities.
• Nominal Roll and Annual Report data; and
• Education studies commissioned by INAC and First Nations; Other documents, data and research, for example, federal disability statistics, studies and reports of the Auditor General and Treasury Board, and recent research on special education and disabilities.

Key Informant Interviews

• National and regional INAC officials with responsibilities related to SEP (n = 13) National and regional First Nation authorities (n = 19)
• National and regional INAC officials with responsibilities related to SEP (n = 13)
• First Nation and First Nation-affiliated educators, administrators and support personnel (n = 40)
• Academic researchers (n = 13) Federal officials (INAC and other departments) with particular knowledge related to Aboriginal education and Aboriginal disability issues (n = 6)
• Academic researchers (n = 13) Federal officials (INAC and other departments) with particular knowledge related to Aboriginal education and Aboriginal disability issues (n = 6)
• Provincial government officials with particular knowledge of special education programming in provinces, standards, recent research and evaluations (n = 16)

Site Visits

• Site visits in Alberta, British Columbia and Quebec (n=3) organized with the assistance of First Nation educational authorities.

An evaluation advisory committee comprised of INAC and First Nation representatives, drawn from the Program's national level First Nations Working Group, provided input into the evaluation's Terms of Reference (approved at the March 2007 meeting of the Department's Audit and Evaluation Committee). This committee also provided advice on the evaluation's interview questions, documents, research and academic experts of interest, and on a draft of the report's conclusions, recommendations and management response.

Over the course of the research, the Evaluation manager had two discussions about the evaluation's progress with the Chiefs Committee on Education (CCOE) and the National Indian Education Council (NIEC) of the Assembly of First Nations (AFN), as well as one discussion with educators affiliated with the First Nations Education Council in Quebec.

3.3 Research Limitations

Because of data and time limitations, it was not possible to research the delivery of high- cost special education services to First Nation students on reserve who are attending provincial schools, except broadly. Several provincial ministries indicated that they are in the process of evaluating their programs.

Although First Nation stakeholders provided input into the evaluation, it was not possible to reach as many participants as originally planned. While representatives from all four of the full management FNRMOs were interviewed, for example, representatives from only six of the 14 partial FNRMOs were interviewed. Input from these organizations via SEP's Annual Reports was also limited (only 8 FNRMO reports were reported as being included in the 2005-06 Annual Report roll-up).

In the end, the evaluators held interviews with school level personnel from three provinces (B.C., Alberta, and Quebec); more than one half of those contributing came from B.C. No First Nation students or family members were interviewed as part of the study. As such, the views recorded in this report should be considered as representing the voice and experiences of key informants rather than the views of a representative sample of SEP stakeholders.

Part II: Evaluation Findings

4 Evaluation Findings

4.1 Needs and Contributing Factors

In general, there has been marked growth in demand for special education programming in Canada over the past 20-30 years.

In Manitoba overall, from 2000 to 2005, the number of special education elementary and secondary school students grew by 22.4% and special education costs grew from 14.8% to 16.3% of total education spending, an increase of $21 million annually. [Note 8]

The Government of Saskatchewan has seen total school enrolments go down by 12% over the past 10 years, but cases of students requiring special education rise by 48%. [Note 9]

In British Columbia, the number of high-cost special education students attending provincial public schools rose by almost 2,000 students between 2001-02 (17,842) and 2005-06 (20,885). [Note 10] The growth rate of some higher cost funding categories rose substantially (at the high end were students reported with Autism), while some others decreased in size (e.g., students with physical dependencies). Part of the growth in student numbers can be attributed to changes in the definitions of some funding categories (e.g., those governing the identification of students with autism were altered to align with provincial health sector guidelines).

These trends are not unique to Canada. In the United States, the number of special education students as a percentage of all elementary and secondary school students increased from 7.5% in 1976 to 12.2% in 2004 (The American Centre for Special Education Finance also reports a 'dramatic' increase in federal government spending on special education, from $4.3 billion in 1999 to $10.1 billion in 2004). [Note 11]

Researchers say one reason for this trend has been medical breakthroughs that have prolonged the lives of children with special needs. Another explanation is growing awareness of and attention to people with disabilities and increased capacity to identify and address the needs of people with disabilities.

However, there is much still to be learned about the level of need for special education, both low and high-cost, in general, and in First Nation communities, in particular. Statistics Canada, in a study published early in 2007, asserts that little is known about the current prevalence of students with disabilities in general across Canada, the educational services they receive, the proportion of children who are receiving support, or the benefit of the support they are receiving. As a result, this agency is calling for greater attention and research in the area. [Note 12]

Exacerbating the gaps in knowledge of special education generally, Aboriginal students' needs are even less well documented at the national level. Since SEP was implemented there have been some advances in knowledge, due in part to program's support, but overall, knowledge and research on special education needs in First Nation communities remain limited and inconclusive in terms of the cultural aspects of special needs.

Part of the challenge for research and data collection stems from the relatively small size and dispersion of students with special needs on reserve (e.g., small numbers of students must be suppressed for privacy considerations). New, more recent national level secondary data should be available over the next year or so, but was not available in time for this evaluation. Coverage is another issue - Canada's key source of statistical information on disabilities, Statistics Canada's Participation and Activity Limitation Survey (PALS) does not cover the on-reserve population (or populations in institutions).

Finally, the information available on children and youths living with disabilities does not always refer to levels or specific types of needs or disabilities (SEP's Annual Reports and INAC's Nominal Roll, for example, do not capture this type of information). This is a challenge which affects our level of knowledge about disabilities across Canada, and is in part due to differences in how disabilities are defined. According to a 2003 study supported by HRSDC, definitions of disability vary widely across the country, even within federal government agencies, across provincial education ministries and school jurisdictions. [Note 13]

The research which is available on Aboriginal and First Nations communities strongly suggests that disabilities and special needs are high on reserve, in fact considerably higher than in the population at large. A recent literature survey commissioned by Human Resources and Social Development Canada, [Note 14] for example, estimates the rate of disability in the Aboriginal population as at least double the rate in the overall Canadian population and in some places three-to-five times the average, depending on the community and on the specific disability.

The National Aboriginal Health Organization's (NAHO) First Regional Longitudinal Health Survey (RHS), supported by the First Nations and Inuit Health Branch of Health Canada, also found evidence of higher than average rates of disability in 2002 and 2003: [Note 15]

'.the rate of disabilities among First Nations children is almost double that for Canadian children in general (7.8% versus 4.4%), considering only those disabilities asked about in both the RHS and the Canadian National Health Survey.' [Note 16]

In British Columbia, a recent First Nations School Association/First Nation Education Steering Committee (FNSA/FNESC) commissioned evaluation found that 30% of 5,608 students were identified as having moderate to severe special education needs in its survey of 106 Band-operated schools (out of a possible 123 schools). [Note 17] These needs were identified by the schools in line with a pre-established set of criteria established for the evaluation rather than through assessments. (The exercise is being used by the FNRMO as a basis for further examination of patterns of disabilities across schools, disability types, ages and grades).

Of the 1,672 students identified (N=5,608), about 56% had previously been assessed formally, and the remaining 44% informally (In this respect it should be noted that FNSA/FNESC has also commissioned an annual series of psycho-educational assessment exercises since the late 1990s (see also section 4.2.3).

A 2001 survey based research study conducted by Fred Wein and Isabel den Heyer, commissioned by the Mi'kmaq Kina'natanewey (the education authority for 9 of 13 Mi'kmaq communities in Nova Scotia), reported on the types, severity and prevalence of special education needs of Mi'kmaq students in those communities. [Note 18] The researchers found that 35% of students in these communities had moderate to severe special education needs, compared to 10-15% in the mainstream North American population and 17% in Nova Scotia. They said their estimates were consistent with those reported in other studies they had found, that is the proportion of Aboriginal students needing special education is two to three times the proportion for the mainstream student population.

Bearing in mind that the study was conducted prior to SEP's implementation, the researchers observed that the proportion of Mi'kmaq students needing special education services was higher in schools off-reserve than on-reserve. They also noted that males were more often affected, and that the need for special education services was mostly due to learning problems, particularly with reading and writing, and emotional/behavioural problems.

They also reported that about a quarter of students were at risk of failure. The most common reasons given for this were poor attendance, lack of motivation, behaviour problems, instability in the family, lack of family support, academic deficits, and student health problems. During the evaluation, stakeholders also raised concerns about the challenges of working with high proportions of students in need, at all levels, and also raised concerns about the uneven quality they were seeing in some assessments which could be contributing to incorrect diagnoses of needs.

In summarizing the findings of ten case studies on successful Aboriginal schools in Canada, published by the Society for the Advancement of Excellence on Education, [Note 19] researcher George Fulford reported:

Special education is a particularly vexing challenge for First Nation educators, where the level of low-needs behavioural and academically-delayed students is two or three times the level estimated for the general population.

While further research needs to be done on the reasons for this. we believe that the high numbers of such students may be one of the major factors responsible for the achievement gap identified between Aboriginal and non-Aboriginal students across Canada. Further research is also needed to determine what portion of that achievement gap is attributable to instructional deficits rather than learning disabilities.

Some studies identify areas in which aboriginal students appear to be over-represented. Annual reports from the British Columbia Ministry of Education say Aboriginal students have been overrepresented in the Behaviour Disabilities group for several years and the number of Aboriginal students in the Learning Disabilities group has been increasing. [Note 20] External studies commissioned by First Nations, and several First Nation and INAC key informants in British Columbia have noted over-representation in the 'learning disabilities,' 'serious behaviour' and 'multiple disabilities' categories.

Data from the NAHO survey (2002-03) suggest that First Nations children living with disabilities are more concentrated in small isolated rural areas than elsewhere, but the results are not statistically significant (see footnote 15 for further details). Preliminary research on special education needs in First Nations schools commissioned by FNSA/FNESC in B.C. established findings which suggest there were variations in rates across communities, with greater concentrations of students with disabilities in smaller (more isolated) schools. Many recent studies, in part supported by HRSDC and Health Canada, have recently been looking at issues of access for children, families and adults with disabilities, citing particular gaps in northern, rural and more isolated locations, and distinct challenges in addressing these gaps due to cultural and linguistic differences.

Because the proportion of children and youth in the on-reserve population is high (in 2004, children aged 0-14 represented 34.3% of the on-reserve population and 15-29 year olds, 26.2%) [Note 21] the need for special education programming is likely to remain high unless disability rates and special education needs, and/or the costs of meeting student needs are reduced over time.

In this context, it should be noted that early intervention, both before children enter school and during their early school years, as well as prevention programming are seen by academic experts and practitioners as key to: improving student outcomes and improving the well-being of disadvantaged and at-risk children with disabilities and their access to effective learning opportunities; and to reducing many disabilities over time. The federal government-funded Centre of Excellence for Children and Adolescents with Special Needs, for example, has concluded that: '. Prevention and early intervention are of vital importance because they provide the best prospects for improving children's health and education." [Note 22]

4.2 Early results

The data in this section provide a profile of HCSE students as portrayed by Nominal Roll and Annual Report data. These two systems are the key means by which INAC captures information on SEP students (see also section 5, Performance Monitoring).

As discussed in greater detail in Chapter 5 (Performance Monitoring), these data sources do not, at present, allow for a clear, accurate or consistent understanding of the numbers of students served by the Program or awaiting services.

The two systems do concur, however, in suggesting that significant inroads are being made in many regions in identifying and supporting First Nations students living on reserve who require high-cost special education programs and services. In addition, data from the Annual Reports also indicate that gains are being made in assessing students' learning needs and developing Individual Education Plans, but that there remain gaps in coverage in these areas.

4.2.1 High-cost special needs student identification

The following table shows the numbers of students identified on the Nominal Roll as supported by the Special Education Program since the Program began up to 2006-07. As shown below, the number of high-cost special education students identified by schools increased from 7,596 in 2002-03 to 12,730 in 2006-07.

Source: Nominal Roll

According to the Nominal Roll, the distribution of identified HCSE students varies widely by region, from 2.3% in B.C. to 22.5% in Alberta for 2005-06:

Source: Nominal Roll, 2005-06
Note: Grand total includes 50 non HCSE students identified as "Other/Yukon"

Further analysis of Nominal Roll data by geographic zones (up to 2005-2006) was not pursued due to a preliminary review indicating significant presence of non-coded entries.

The following table shows the numbers of students identified by Annual Report Data as being on the Nominal Roll. This data differs from the picture presented by the Nominal Roll, above. The Annual Reports suggest that the number of students supported by SEP rose in 2004-05, but dropped in 2005-06. The evaluation research suggests that this drop may in part be due to gaps in reporting and the use of non-standard reporting formats. However, reports for 2006-07 data were not available to the evaluation to validate this hypothesis.

Source: See Footnote [Note 23]. (*INAC, Education Branch, Amendment, December 2007). Note: Data for 2003- 04 includes First Nation schools only.

Some of the differences in numbers between the two data systems can be attributed, firstly, to the fact that data from one of the two FNRMOs in Quebec (FNEC) has not been included in the Annual Report Roll-ups for most of the Program's implementation. This was due to a differing report format which was approved on a pilot basis (In 2004-2005, for example, the roll-up does not take into account some 985 high cost students in Quebec). Data from that organization's 2005-06 Annual Report was also not included in the 2005-2006 Annual Report roll-up, however, figures pertaining to the Annual Reports for that year have been amended to reflect corrected figures provided by the Education Branch.

Secondly, Annual Reports of non-Band schools were not initially included in the annual roll-up. Other contributing factors include differences in the census dates of the two systems, differences in coverage (e.g., the Nominal Roll does not capture some self- governing communities), the use of differing Annual Report templates, changes to the reporting questions, and non-reporting issues in both systems (See also Chapter 5, Performance Monitoring).

For 2003-04 some of the differences in numbers between the two data systems can be attributed to the fact that Annual Reports of non-Band schools was not included in the annual roll-up. Other contributing factors include differences in the census dates of the two systems, differences in coverage (e.g., the Nominal Roll does not capture some self- governing communities) and non-reporting by SEP recipients in both systems.

HCSE students by type of school Nominal Roll data show that in 2006-07, 22% of SEP students attended provincial schools, 75% attended band-operated schools, and 3% private or federal schools:

Source: Nominal Roll. *'Other' includes federal and private schools.

The following table shows the distribution of students by type of school according to the SEP Annual Reports for the year 2005-06. This data suggests that 95% of SEP supported students attend First Nation schools and 5% non-band operated schools (e.g., provincial, federal and private schools).

Source: See Table 4.3 source (2005-06). (* INAC, Education Branch, Amendment, December 2007).

SEP Annual Reports do not provide information on the distribution of high cost special needs students at the sub-regional level, by sex, by age or grade level. The Nominal Roll does allow for this type of disaggregation, as discussed in the following sections.

HCSE students by gender

The Nominal Roll data show that approximately two-thirds of identified HCSE students are male:

Source: Nominal Roll

Regions/grades of HCSE students

A breakdown of Nominal Roll data by region and grade in 2005-06 shows the highest number of high-cost special education students between grades 3 and 10. The 573 "SS" students receive "special services," that is, they are not in a specific grade.

Source: Nominal Roll

4.2.2 Province of B.C. data on First Nations special education students who attended provincial public schools in 2004-05

A report from British Columbia on the number, special education category and cost of services/student in 2004-05 gives a sense of student needs. The following table shows the number of high-cost and low-cost special education students, registered on the Nominal Roll, who are registered in provincial schools. Approximately 33% were high cost special education. The special education categories with the greatest numbers were intensive behaviour interventions/serious mental illness, physical disability or chronic health impairment, and moderate to profound intellectual disability.

Source: Synthesized from Province of British Columbia, Aboriginal Students Accessing Special Education, Nominal Roll, September 2004 (By Band, Special Needs Reporting). Mimeo provided to AES by INAC British Columbia, March 2007.

4.2.3 Student Assessments and Individual Education Plans

Data on student assessments, students waiting for assessments, and students with Individual Education Plans has been drawn from Annual Reports rather than Nominal Roll. This is because, unlike the Nominal Roll, the Annual Reports address these issues explicitly. However, as earlier noted, it is not possible to make comparisons between data in the two systems, as they are based on significantly different numbers of HCSE students (e.g., Annual Reports count some students who are not required to report on the Nominal Roll while not all schools provide Annual Reports).

Student Assessments: Annual reports indicate that several thousand student assessments have been conducted each year since the program started: 4,222 in 2002-03, 6,485 in 2003-04, 6,697 in 2004-05 and 5,674 [Note 25] in 2005-06. Key informants reported many challenges finding specialists to conduct assessments, especially in more isolated areas, and also indicated that the quality and accuracy of assessments can be uneven. [Note 26]

Backlogs are also reported in some areas because of a shortage of qualified professionals. One key informant said the City of Winnipeg school district, serving roughly the same student population as the population of Manitoba First Nation schools, has 58 speech pathologists on staff, compared to three on the FNRMO staff. Key informants and the literature suggest that shortages of professionals, are also affecting non-aboriginal students across the country to varying degrees.

Partially in recognition of such backlogs and gaps, INAC sought (and received) approval to adopt the intervention approach so as to ensure students were not left without additional support while waiting for formal assessments (it is too early to see what impact this might have on serving students as four regions have just started to implement the approach over the past year).

In order to address the demand for formal assessments, FNSA/FNESC in British Columbia has commissioned specialists to conduct annual province wide assessment campaigns for several years. During 2005-06, 144 such assessments were conducted (100 were supported the FNRMO, and the remaining 44 by individual schools and communities). At the end of this campaign, the lead specialist indicated there were still some 400 students waiting for assessments. [Note 27]

It is not known whether the assessments reported reflect assessments of newly identified students or follow-up assessments of students who have already been identified as HCSE, since guidelines suggest periodic reassessments are important (The most recent version of the Annual Report does provide a means to distinguish between new assessments and reassessments, but was not in use during the time period studied). One question raised by the data is whether there are students who are found through assessments not to need HCSE services; it appears from the Annual Report form and report data that every child that is formally assessed is subsequently added to the list of HCSE students.

Source: See sources for Table 4.3. (*INAC, Education Branch, Amendment, December 2007). Note: 2002-03 data does not include Quebec. 2002-03 / 2003-04 data includes only First Nation schools.

Based on a calculation of the difference between numbers of students referred for assessments and those in receipt of assessments, data from the Annual Reports suggest that a significant number of students are not getting assessed during the year in which they are referred for assessment, as shown below (Table 4.10).

Source: See sources for Table 4.3. (*INAC, Education Branch, Amendment, December 2007). Note: see note, Table 4.9.

Starting in 2004-05, the Annual Reports began asking the number of reassessments required for students with high cost special needs. In 2004-05, 2,931 students were reported as requiring reassessments, and in 2005-06, 2,756. [Note 28] It is not clear whether these reassessment requirements are included in the numbers of students referred for assessments or not. The last version of the Annual Report (revised in 2006) includes a revision which should improve clarity on this issue for the future.

The Annual Report questions also capture data on students whose needs were not met (partially or at all) and on the numbers of these students who have not been assessed. According to this information, 1,105 students whose needs were not met in 2002-03 were without assessments, 1,405 in 2003-04, 1,627 in 2004-05 and 1,287 [Note 29] in 2005-06. It is not clear from the data, whether these students were referred for assessments during the current year or not.

Individual Education Plans: Annual Reports show that many but far from all HCSE students have Individual Education Plans:

Source: See sources for Table 4.3. (*INAC, Education Branch, Amendment, December 2007). Note: see note, Table 4.9.

* Column totals do not add up to source totals (they add up to 5,720; 7,005; 9,722).

The data in the following table shows that a substantial number of students who have been assessed as having high-cost special education do not have Individual Education Plans.

Source: See sources for Table 4.3. Note: see note, Table 4.9. * Column total does not agree with source totals (the column adds up to 885). **ICEM only.

While the SEP reports do not probe for the completeness or quality of Individual Education Plans, Key Informants reported many challenges in developing Individual Education Plans, let alone good or complete plans, particularly in more isolated areas.

4.2.4 SEP Expenditures

Public Accounts records show that the department spent slightly more ($1,451,200) than its total SEP allocation for the period reviewed. However, according to INAC, these figures are overstated by $1,972,262, due to a difference between OASIS and Public Accounts in 2004-05 arising from the inclusion of Employee Benefits Program and the incorrect reconciliation of Flexible Transfer Agreement expenditures in the Regions to the Special Education Special Purpose Allotment. Over the course of the evaluation, evaluators also learned that the Department had a process for moving funds between regions to ensure that no funds would lapse.

Source: Public Accounts.

As shown below, there is considerable variation across regions when SEP expenditures are divided by the number of beneficiary students reported no matter whether the Annual Report or Nominal Roll is consulted. There also is considerable variation in the cost per student depending on what data source is consulted:

Sources: Financial data, see Table 4.13, Nominal Roll (2005-06), Annual Report: see Table 4.3. (*INAC, Education Branch, Amendment, December 2007).

The evaluation research also suggests that there is some hidden cost pressures that are not identified in the numbers reported on either the Nominal Roll or Annual Report. Provincial records provided to the evaluators by INAC for one province in 2004-05 show that close to 500 students supported by the Special Education Program are not identified in either data system.

Moreover, this information also indicates that, in addition to the $4.3M provided to the Province by INAC in support of First Nations students living on-reserve who attend provincial schools, this province invested an additional $3.7 million in the education of these students (in line with the published funding provided to students assessed as required services according to differing special needs categories).

This gap in reporting, as well as evidence of the use of 'own resources' represents a significant cost pressure on the Department's resources. The Department's decisions on future allocations should be able to take such information into account. According to INAC's 2007-08 Budget Management Regime, for example:

"It will be important to ensure that all students who are receiving services from the SEP are coded in the nominal roll... This is a basic requirement of the program and must be adhered to. Without the necessary nominal roll data, SEP allocations for future years will be reconsidered."

4.2.5 Full Service and Partial FNRMOs

An innovative feature of SEP's delivery involves the use of First Nations Regional Management Organizations (FNRMOs) to deliver second level services and administrative support to First Nations schools and communities.

INAC's expectations for the FNRMOs appear consistent with the federal government's expectations for minority language school boards. [Note 31] According to an evaluation commissioned by the Department of Canadian Heritage, these Boards, which the government helped create, were proving instrumental in achieving federal policy objectives in settings often characterized by dispersed or isolated schools with small populations.

SEP's FNRMOs vary across a number of parameters (see Table 4.15, below). Firstly, their catchments differ significantly. Two provinces are served by one FNRMO each: in Manitoba, MFNERC close to 60 schools; and in British Columbia, FNSA/FNESC serves between 125 to 130 schools (school numbers can fluctuate depending on their enrolments).

Source: Education Branch, received by AES on November 5, 2007.
Note: There are no FNRMOs in Ontario.

Most other regions are served by more than one FNRMO. Alberta's four FNRMOs are aligned with Treaties, 6, 7 and 8; they serve between 8 to16 schools each (The numbers of FNRMOs in this province have fluctuated since SEP began). .

In Saskatchewan, FNRMOs are aligned with Tribal Councils. The largest of these, PAGC, serves 28 schools which are attended by about 40% of the on-reserve population. In Nova Scotia, the MK serves 10 self-governing schools, and in New Brunswick, the First Nations Education Institute Inc. also serves about 10 schools.

In line with Departmental expectations that these organizations would gradually advance towards the full management of SEP in their catchment areas, the FNRMOs also differ in terms of their mandates. When SEP was initiated, only two FNRMOS held full management responsibilities, FNEC and FNSA/FNESC.

The Program's extension in 2005 set the stage for the approval of several other full management FNRMOs, although to date only two more organizations (PAGC and ICEM) have achieved that status.

SEP's performance measurement strategy does not place significant emphasis on tracking the development of FNRMOs. The only performance indicators which mention the FNRMOs directly, for example, reference the numbers of First Nation schools associated with these organizations. While the Annual Reports do provide for an overview of the types of services provided by FNRMOs, many of the questions do not lend themselves to assessing results over time. For example, while the report asks FNRMOs to check off the types of organizations they collaborate with, it does not ask the nature of the collaboration, or reasons for which linkages are not occurring. Moreover, not all FNRMOs appear to be represented in the national roll-ups of SEP's Annual Reports (only 8 of a possible 18 FNRMOs were represented in the 2005-06 roll-up of SEP reports).

According to SEP stakeholders, First Nations are interested in second-level services when they see the benefits of the support received, often in conjunction with assessments and Individual Education Plans, professional development, and in promoting early intervention, family and community support. Some of the beneficial initiatives noted include:

• Concentrated training efforts in Saskatchewan towards the development and implementation of results oriented Individual Education Plans (in part as a result of a pilot initiative in Saskatchewan and Manitoba between First Nations and provincial governments);
• Efforts towards the establishment and implementation of systemized electronic student tracking systems in Nova Scotia and Quebec;
• The development of annual assessment programs or campaigns to meet the need for assessments (in B.C). By mid-2006, six campaigns had been organized. During the most recent, 144 psycho-educational assessments were conducted by a team of seven specialists;
• The development of approaches and techniques to address distance and isolation. These include: the employ of traveling specialists; the hiring of local staff; telephone support or hotlines to assist schools in a timely manner; and the use of video- conferencing and other electronic technologies; and
• The encouragement of results based work plans or proposals to help strengthen planning around special education and to maximize the use of SEP funds.

Stakeholders, however, also identified some of the challenges they are experiencing in serving the needs of a particularly vulnerable target population. Key Informants, particularly those working in more remote areas or with smaller FNRMOs identified difficulties in attracting and maintaining staff in competitive environments, finding or negotiating with other organizations for support, and the need for mentoring and support in order for them to become effective.

Stakeholders also identified the following issues which they identified as important to take into consideration in order to support the growth of FNRMOs:

• The potential benefits of accessing second level services are not always or not yet obvious to school communities;
• Being at arms-length from political organizations, yet having buy-in from schools, communities and leaders, is seen as important, in part to enhance the stability of FNRMOs, and in part, to enhance neutrality in funding decisions; and
• The processes through which FNRMOs become full service operations are not clear (approval rests with the differing regions).

The Canadian Heritage evaluation, referred to earlier, found that the stability of the Minority Language School Boards owed a great deal to the federal government's long- term investments in their growth, and recognition that such Boards faced additional and ongoing supplementary costs in fulfilling their mandate. These supplementary costs were seen to arise from the fundamental issue facing minority school boards, which, as mentioned previously, concerns the challenge of

'.offering quality education relatively equal to that of the majority system in an environment in which the lack of a critical mass combined with problems specific to a minority system systematically lead to higher operating costs.'

The evaluation found that the stability of the Boards should not be considered assured without continued support from the federal and provincial governments, concluding that the extent to which the supplementary challenges should be identified, quantified and funded would in large part determine the stability of minority school boards over the long term.

4.2.6 Culturally sensitive services

Some key informants said some standard tests had been adapted to allow for cultural sensitivity, but none had been specifically designed for Aboriginal students. However, they thought the specialist's skill and sensitivity more important than test adaptations. The possibility of students being incorrectly identified as special needs because of cultural differences was also raised.

4.2.7 Teacher training

SEP plans were to support professional development for school staff, to increase the number of teachers and paraprofessionals in First Nation schools with provincially- recognized special needs accreditation.

There have been increases in the number of teachers identified as certified or qualified for special education, but no way to assess whether these increases are adequate.

The Annual Reports also indicate that many training sessions have been provided, however there is no information about their success. Anecdotally, it seems that there is greater awareness in schools about the need for Individual Education Plans. According to Key Informants, the plans have been improving over time, and there is more information- sharing and networking between teachers and schools. Some organizations have noticed, through observations and testing, early indications of improvement in student attendance, behaviour and academic performance.

However, stakeholders continue to report capacity gaps in schools and a pressing need for more professional development and training for teachers and paraprofessionals, including, according to several educators interviewed, greater attention to special education for teachers in training. Many also noted problems with the retention of qualified or trained staff.

A Council of Ministers of Education (Canada) observation on the urgent need for professional development holds true for schools serving First Nation students.

... the greatest responsibility for implementing the policy of inclusive education falls to the classroom teachers, resulting in greatly increasing demands on their time, attention and flexibility.

Because students with special needs may be in every classroom, the challenge for the school boards and educational authorities is planning and supporting those classrooms and teachers so that student diversity is valued, the potential of all students is realized, and teachers have the necessary assistance. Professional development is a crucial component of this support. [Note 32]

4.2.8 Parental Involvement and Community Outreach

Anecdotally, there are hints that awareness of First Nation families and communities about special education generally, and the needs of special education students, is increasing. This may continue to grow with community awareness-raising efforts by FNRMOs and schools seeking discussions with parents and elders about Individual Education Plans for particular students.

Many of the educators with whom the evaluators spoke saw parental support and community participation as key to improving outcomes for First Nation students living on-reserve with high-cost special education needs. Many noted that they were taking steps to encourage involvement and seeing improvement in parental involvement, but also spoke of the challenges still being encountered in encouraging family support and changing attitudes towards education and disabilities.

According to SEP's 2005/06 Annual Reports, the proportion of schools reporting parental involvement in consultations ranged from 73% and 78% in B.C. and Saskatchewan, respectively, to 98% and 100% in Manitoba and the Atlantic. The proportion of those reporting involvement with Individual Education Plans ranged from 79% in Saskatchewan and 80% in B.C, to over 90% of schools in most other regions (Note: Annual Report aggregation indicates that no parents in Quebec were involved in the development of IEPs, however, a review of the reports submitted by the two FNRMOs indicates that all schools, including Quebec, reported parental involvement in this area).

Given that the response options in the Annual Report form do not permit more than a simple yes or no, it is difficult to know the extent to which parents in any given school community participating, or the degree to which their involvement may be improving over time.

4.2.9 HCSE student learning outcomes

There have been some efforts by INAC to collect student outcome information at the national level through the Annual Reports. However, performance measurement has been limited, focusing not on outcomes but on whether IEP goals have been achieved or not, and on annual achievements at the aggregate level rather than on longer-term tracking (e.g., cohort tracking or individual progress/transitions through levels). In addition, due to variations in the annual report questionnaire, the information which has been gathered is not comparable over time.

At the same time, the evaluation research found examples of First Nation efforts at the regional level to capture more relevant and comprehensive information on student outcomes. These include, for example, a pilot project, known as the Student Outcome Rubric, initiated by the Western and Northern Canadian Protocol (WNCP) and involving the Federation of Saskatchewan Indian Nations as well as Saskatchewan Learning and Manitoba Education and Training. [Note 33] In Nova Scotia, the Special Education Policy of the Mi'kmaq Kina'matenewey indicates that their interest in this approach was due in part to its consistency with the Province's approach to examining student outcomes. In British Columbia, FNSA/FNESCs annual published reports indicate that various sources are used to report on student outcomes (including standardized tests or assessments, school records and anecdotal information) across a number of measures (e.g., academic performance, improved attendance/behavior, child development), as well as to measure other outcomes including improved community awareness, parental involvement and school capacity.

5 Performance monitoring

As earlier noted, the Department has two main sources of Special Education Program data and information: the Nominal Roll and the SEP Annual Reports. [Note 34]

5.1 Nominal Roll

The Nominal Roll system is a database which captures basic information about students and services received through an annual census (September 30th of each school year). In general, this census information is used by INAC to support program funding and governance, specifically: to monitor the activities funded under the education program; fulfill reporting and accountability requirements; and, develop and evaluate proposals for policy, programming and funding. [Note 35]

The National Program Guidelines for the Special Education Program (2006) say that recipients (i.e., organizations providing Nominal Roll reports) must demonstrate the eligibility of a student to be included on the Nominal Roll before becoming eligible for funding.

To be eligible for inclusion in the Nominal Roll, a student must be: [Note 36]

• "Enrolled in and attending a federal, provincial, band-operated or private/independent school recognized by the province in which the school is located;
• "Aged 4-21 years (or the age range eligible for elementary and secondary education support in the province of residence) on December 31 of the school year in which funding support is required, or a student outside of this age range who is currently funded by INAC for elementary and secondary education; and
• "Ordinarily resident on reserve. Ordinarily resident on reserve means that the student usually lives at a civic address on reserve, or is a child in joint custody who lives on reserve most of the time, or is staying on reserve and has no usual home elsewhere. Students continue to be considered ordinarily resident on reserve if they return to live on reserve with their parents, guardians or caregivers, even while attending school or working at a summer job. In this context, reserves are deemed to include land set aside for the use and occupancy of an Indian band, along with all other Crown lands that are recognized by INAC as settlements of the Indian band of which the student is resident.'

The Nominal Roll tracks the number of high-cost special First Nation education students that live on reserve, the schools they attend (provincial, band-operated, private, federal) and their community. SEP's Terms and Conditions say the Roll is also to track costs of HCSE services identified by Individual Education Plans based on maximum amount payable guidelines.

5.2 Annual Reports

Annual Reports are lengthy questionnaires that all FNRMOs and schools that receive SEP funding must submit. There have been several iterations of the forms since SEP was launched (The initial (2002) and revised (2006) National Program Guidelines contain only two of the variations).

INAC's Recipient Reporting Guide (2007-08) National Template identifies May 1^st as the census date. SEP's National Program Guidelines, however, do not specify a census date, only a submission date. This date has varied over time. Currently all reports are due by the end of July. This means that data on supported HCSE students could (and is) collected by schools at differing points during the year.

The reports' purpose is to make the case for additional federal funds for high-cost special education services (one version of the form states: "INAC continues to request from Treasury Board increased funding to provide services to special needs children that are at least equivalent to that provided by provincial education authorities. The data below is required to support that request").

The forms also serve to identify students served, assessed and with Individual Education Plans, outcomes (through the achievement of IEP goals), track services delivered (to students and to schools) and operational issues (e.g., schools with Special Education policies, how often the schools review Individual Education Plans, as well as practices to engage parents and caregivers).

Prior to a major revision to the Annual Report in 2006, one of the earlier versions of the FNRMO report had more than 180 questions that were difficult to follow and would require considerable time to complete. [Note 37]

It asks the number of students that were and were not listed on the previous September's Nominal Roll report, and the number of HCSE students that were reported and not reported on the Nominal Roll, but makes no request for an explanation of variances.

It asks reasons that services to HCSE students were not provided and then lists nine possible reasons for which this would be so, including the option 'Funding for hiring staff (e.g., TAs, teachers, etc).' Respondents are then asked to estimate the funds necessary to fill the funding gap. In its report for 2005-06, one FNRMO wrote in text to suggest that the FNRMO also had unmet needs (this FNRMO added the text: "No funding for RMO support staff, executive assistant, Data Analyst, High Cost Spec. Ed. Service Providers (Psychologist, Speech Language Pathologist)" as one of the first response options.)

The October 2006 version of the SEP FNRMO and School Annual Reports show that efforts have been put towards streamlining the forms and bringing them more into line with the intervention model of delivery. However, there remain over 130 questions (including sub-questions) in the FNRMO report, many of which still appear unclear, redundant and time consuming to complete. Consider the following 18 questions which revolve around the identification of and support for students with high cost special education needs, assessments and Individual Education Plans:

• How many eligible students in the schools are RECEIVING Special Education Services from SEP funds? (Q 1a)
• How many eligible students in the schools receiving Special Education Services from SEP funds HAVE AN IEP? (Q 1b)
• How many eligible students in the schools are NOT RECEIVING Special Education Services from SEP funds? (Q 2a)
• How many eligible students in the schools NOT RECEIVING Special Education Services from SEP funds have an IEP? (Q 2b)
• How many eligible students in the schools have been IDENTIFIED THROUGH FORMAL ASSESSMENT as having HCSE needs? (Q 3)
• How many students in the schools have been IDENTIFED THROUGH INFORMAL ASSESSMENT as having HCSE needs? (Q 4)
• How many students were REFERRED for a formal assessment within the school year? (# Re-assessments, # New cases) (Q 5)
• How many students were FORMALLY ASSESSED within the school year? (# Re- assessments, # New cases) (Q 6)
• How many FORMALLY ASSESSED students were IDENTIFIED during the school year as having high cost special education needs? (# Re-assessments, # New cases) (Q 7)
• How many students were INFORMALLY IDENTIFIED during the school year has having high-cost special education needs? (Q 8)
• How many students identified as having High Cost Special Education needs and eligible for services under SEP funding DID NOT HAVE ANY OF THEIR NEEDS MET? (Q 28)
• How many students identified as having High Cost Special Education needs and eligible for services under SEP funding HAD THEIR NEEDS PARTIALLY MET? (Q29)
• How many students with High Cost Special Education Needs are in the schools but INELIGBLE for services under SEP? (30)
• Of the students identified in Questions 28 and 29 [BUT NOT 30], How many:
  • Have been assessed by a specialist?
  • Have been assessed through teacher observation/screening;
  • Received a medical diagnosis?
  • Have NOT been assessed? Other (specify)? (Q 31).

Other areas of the current questionnaire are similarly problematic. Questions around training provide little insight into the level, scope or advances in providing professional development. One question asks, for example, the number of teaching staff involved in at least one Special Education activity during the year (Q 15). The range of possible activities included coaching/mentoring to university/college courses, workshops, in school training and conferences or workshops.

While the 2006 version of the SEP Annual Reports (which is not fully compatible with previous versions) was expected to be used for the first time to report on SEP over the 2006-07 period, many institutions used the forms to report on activities for 2005-06 (Some 88 or 21% of all reports submitted for 2005-06 were submitted on the 2006-07 forms). The one FNRMO which had submitted the incorrect template stated that the data it compiled should be treated with caution, in part because it was the first year that it would be completing the SEP annual reports (prior to this time, an alternative format had been used), and in part because of problems in understanding the reporting questions. The use of incorrect forms for reporting year 2005-2006 was identified at INAC's National Headquarters early in 2007 (The annual reporting data is compiled in Ottawa months after the forms are due in the Regional Offices).

Over the course of the evaluation, many interlocutors noted challenges with reporting beyond concerns about the content of the reports. These included concerns that the dissemination of information has been uneven, comment sections have not been aggregated, and that the lack of an electronic format has made both scrutiny and analysis difficult - as has constant changes in reporting questions over the years.

Evaluators were asked to report whether INAC is tracking, measuring and reporting the right information. While Education Branch officials indicate that measures are now being implemented to better track incoming reports, based on the information gathered, it seems that much information is requested and collected at the national level, but that much of it is neither useful nor used.

At the same time, the evaluation research identified examples of First Nations efforts to collect, track and/or make available publicly information and to use it to assist in decision-making and to improve local practice. These include, for example, electronic tracking of individual education plans, results-based workplans or proposals, external evaluation activities, some of which are being adopted across differing regions.

Nominal Roll and Annual Report tracking do not yield clear information on the number of First Nations students living on reserve who require high-cost special education services, whether those students have been assessed and found to need high-cost special education services, whether they have an Individual Education Plan, and whether the Individual Education Plans are being implemented.

Reliable information about whether the HCSE services provided with SEP funding are making a difference for students that receive them is also lacking. There is more emphasis on tracking expenditures and outputs than on the effectiveness of services to teachers and schools and the outcomes for students [Note 38] (Education and Finance branches track SEP expenditures. Because special allotment conditions require the department to return unspent SEP funds to Treasury Board each year, the Department transfers surpluses between regions as years proceed).

In order to better understand whether the program is being properly delivered and whether it is effective, collection of performance information must be organized around five broad questions:

• Is the program reaching the students who need to be reached?
• Is the program providing the services needed by those students?
• Is the program making a difference in the lives and educational outcomes of those
• Is the program building the infrastructure that is needed to meet the needs of teachers who work with those students?
• Is the program raising awareness in communities of the need to prevent those disabilities that can be prevented?

There is a pressing need for an overhaul of reporting forms. Such an overhaul could improve the department's information base and lighten the reporting burden for funding recipients. With proper data gathering tools, the department could even have the capacity to look at cost effectiveness of its programming in different regions and comparability of INAC-funded special education services with those provided by provinces.

This area would benefit from close attention in the current Smart Reporting exercise which is being led by Associate Deputy Minister Neil Yeates.

6 Should SEP stay a special allotment?

The evidence gathered through key information interviews, literature and program document review suggests that, irrespective of the level of integration achieved with regular K-12 programming, SEP should at present remain as a separate funding component of Elementary/Secondary Education authorities.

Some First Nations would like to see high-cost special education as an integral part of the funding formula for First Nation schools (see for example FNEC, 2006:43). Most interviewees, however, indicated that a protected budget, as provided for in SEP's design, had proven to have many advantages in terms of ensuring that the identified funding specifically supported special education programs and services and in terms of understanding the level of need.

If SEP were to be rolled into the overall education program, attention should be given to ensuring that funding conditions and procedures are maintained (and monitored and reviewed) so that SEP funding can be clearly identified, tracked and used only for the purpose of providing special education programs and services.

The numbers of students in need of high-cost special education or the costs associated with providing such services are not yet known to First Nations or to INAC. Funding for the Department's Elementary/Secondary education programs is determined on the basis of population; annually increases are capped at 2%. Should the numbers of First Nations children in need of high cost special education continue to grow, that is, become a larger segment of the school population, the 2% annual increase in funds for overall K-12 programs will not be sufficient to cover needed special education services. As well, given the over-concentration of high cost special needs students within First Nations schools and the impact on regular K-12 programming, a mechanism should be established to review how to fund special education on the basis of needs. Whatever financial controls are put in place, they should be accompanied by good financial reporting.

Given the range and complexity of activities to be supported towards this end, such reporting will also be essential to ensuring INAC and FNRMOs are in a position to evaluate the value-for-money and cost-effectiveness of First Nations special education funding in the future.

Part III: Conclusions, Recommendations and Management Response/Action Plan

7 Conclusions

The full dimensions of the need for high-cost special education services and programs for First Nations students living on reserve are not yet known, but research suggests it is significantly higher than in the general population.

In 2006-07, 12 per cent of all students on the Nominal Roll were reported as receiving INAC supported high-cost special education programs and services, up from 7% of all students in 2002-03. In sum, more than 5,000 more students were reported in 2006-07 (12,730) as having high-cost special education needs than in 2002-03 (7,596).

However, the evaluation findings show that the Nominal Roll and SEP's Annual Reports do not accurately depict the numbers of students being served. There are substantial variations in several regions in the numbers of students being reported on the Nominal Roll and in the Program's Annual Reports. The evaluation also identified several hundred students in one province that do not appear to be represented on either the Nominal Roll or in the Annual Reports but for whom funding is provided.

The Department's capacity to identify, analyze and monitor trends at the national level in First Nation special education is also limited by a paucity of secondary sources of data and research on First Nations students living on reserve with disabilities and special education needs.

Not all students requiring or receiving services have been assessed and/or confirmed as having high-cost special needs, nor can it be said what proportion of these students have Individual Education Plans or are receiving services either of a diagnostician or in special education. Interviews reveal that there are assessment backlogs, particularly in more isolated areas, because of the challenges of finding specialists to do the assessments and, once Individual Education Plans are developed for students, it is challenging for some schools to provide the services they recommend, while the quality of some assessment is also being questioned.

According to the information available, there have been some gains in the capacity of schools and teachers to identify and work with high-cost special education students, and also the capacity of First Nations Resource Managing Organizations to provide support and assistance to schools and teachers. It was also reported that there is increasing awareness in First Nation communities about the importance of special needs education and the need for increased funding.

However, while the evaluation research indicates that First Nations in differing regions are taking steps to collect and report on outcomes and/or have initiated efforts to identify or better understand student outcomes, there is no national system in place for capturing such information. Our analysis suggests that the strongest national data collection system associated with SEP has been the one to monitor expenditures to ensure that the conditions of the Special Allocation are being met. Based on the numbers of students reported in the Annual Reports or in the Nominal Roll, expenditures per student vary widely across regions, and differ as well according to which data base is consulted. The evidence indicates that First Nations in various regions have developed tools and processes to improve reporting and knowledge on special education and that some of these practices are transcending regional borders.

However, despite efforts to institute a national reporting system for SEP, there is no evidence of quality assurance, attention to rigour and integrity of the data at the national level, particularly in the area of comparative year-over-year data collection.

The information available suggests that the number of eligible students requiring high- cost special education programs and services will remain high. This is partly due to the demographic profile of First Nation communities, with children and youths projected to make up a very high proportion of the population. However, the information available also indicates that early intervention and prevention activities help reduce preventable disabilities and improve student outcomes over time.

If the numbers of students requiring high cost special education and the costs of providing services are not clearly understood by the Department, and priority is not placed the quality of delivery (e.g., effective early intervention, pedagogical capacities and family involvement, for example), this area of programming will continue to put considerable pressure on education programming for First Nations as a whole, over time.

8 Recommendations

1. Simplify reporting requirements and focus them around five major questions: Is the program reaching the students who need to be reached? Is the program providing the services needed by those students? Is the program making a difference in the well-being and educational outcomes of those students?
   • Is the program reaching the students who need to be reached?
   • Is the program providing the services needed by those students?
   • Is the program making a difference in the well-being and educational outcomes of
   • Is the program building the infrastructure that is needed to meet the needs of teachers who work with those students?
   • Is the program raising awareness in communities of the need to prevent those disabilities that can be prevented?
2. Ensure all data required of all funding recipients is consistent and adequate for performance measurement requirements.
3. Ensure funding is used only for HCSE services and establish mechanisms to better understand the extent of demand and costs.
4. Strengthen training and capacity-building for First Nation school personnel.
5. Continue supporting mechanisms which will encourage economies of scale, innovation, and the sharing of services.
6. Support prevention and early intervention activities within schools as well as other longer term multi-sectoral efforts to reduce and prevent disabilities on-reserve over time.

NOTE:

At the meeting of the Audit and Evaluation Committee, held December 19, 2007, discussion of the Action Plan and its timing emphasized the importance of relating all Education policy development with, in particular, other evaluations and RMAFs. Data collection and reporting have to address, in a coherent way, the needs of a comprehensive policy approach, which takes into account the impact of specific programs, such as Special Education, on the entire policy suite that INAC is now considering. In this light, the Chair asked that an update on the Action Plan, which follows, be presented at the earliest Fall meeting of the Committee.

9 MANAGEMENT RESPONSE / ACTION PLAN

Project Title: Formative Evaluation of the Special Education Program

Program Project: 05/15

Region or Sector: Social-Economic Policy and Regional Operations Sector

I approve the Management Response / Action Plan

Original signed by:

Appendix A: List of Acronyms

Footnotes:

1. INAC, 2005, Special Education Program Terms and Conditions. (return to source paragraph)
   
   
2. Some self-governing organizations do not complete the Nominal Roll census. (return to source paragraph)
   
   
3. See Section 2.7 for a description of direct and indirect services. (return to source paragraph)
   
   
4. Decisions on SEP funding allocations to schools in areas not managed by FNRMOs usually remain with INAC regional offices. In some regions, for, example, the Atlantic Region, regional office staff and First Nations together review applications. (return to source paragraph)
   
   
5. See section 2.7 for a description of direct and indirect services. (return to source paragraph)
   
   
6. INAC, 2005, Special Education Program Terms and Conditions. (return to source paragraph)
   
   
7. These documents included an external survey-based evaluation recently commissioned by the First Nations Schools Association/First Nations Education Steering Committee (FNSA/FNESC) in British Columbia (Stan Auerbach, 2007, Special Needs Students in First Nations Schools: Inclusion in School Based Special Education Programs, submitted to First Nations Education Steering Committee and First Nations Schools Association (FNESC Education Evaluation Report)), and a special submission to the evaluation prepared by Quebec's First Nations Education Council (FNEC) (FNEC, 2007, Historical and Comparative Background and Position Paper on FNEC Special Education Program, prepared in the context of the national SEP evaluation). This report traces that organization's involvement in special education since the 1970s and provides comment on the SEP itself. (return to source paragraph)
   
   
8. Based on Ministry of Education, Citizenship and Youth, Research and Planning Branch, A Statistical Profile of Education and Training in Manitoba, 2000-01 to 2004-05 (http://www.edu.gov.mb.ca/strategy/statprofile.html)  . (return to source paragraph)
   
   
9. Government of Saskatchewan communication (confirmation September 18, 2007). (return to source paragraph)
   
   
10. British Columbia, Ministry of Education, 2006, Students with Special Needs "How Are We Doing? Province /Public Schools Only, pp.2-3 (To note the number of special education students not identified as requiring additional funding decreased between 2002-02 to 2005-06). (return to source paragraph)
    
    
11. Thomas Parrish, 2006, National and State Overview of Special Education Funding. (return to source paragraph)
    
    
12. Sharanjit Uppal, Dafna Kohen and Saeeda Khan, 2006, Educational Services and the Disabled Child, Statistics Canada  . (return to source paragraph)
    
    
13. Human Resources Development Canada, 2003, Defining Disability: A Complex Issue, . (return to source paragraph)
    
    
14. Jamie MacDougall, 2006, Where the River Flows: Aboriginal People with Disability, A Literature Review: Focus on Employment, prepared for HRSDC, Aboriginal Affairs Directorate, October 2006. (return to source paragraph)
    
    
15. Findings were based on a total of 22,602 surveys collected from 238 First Nation communities in 10 regions across Canada. The survey discusses incidence rates in terms of specific disabilities, but not in terms of differing levels of severity.
    
    An independent review of the RHS was conducted by the Harvard Project on American Indian Economic Development (Harvard Project on American Indian Economic Development (2006), Review of the First Nations Regional Longitudinal Health Survey (RHS) 2002/2003, April, 2006. The review concluded that in general, it was '…impressed with the overall quality of the 2002/2003 RHS, its consistency with previously validated survey research practices and its innovations with respect to the involvement of First Nations communities and their representatives in the research process.' The review found that the survey's sample size was adequate for cross-sectional and longitudinal analysis, that there had been a high response rate, and a high level of community representation. Among the limitations noted were that the sample size was insufficient for data analysis among smaller communities. The RHS team is currently planning for a second phase RHS   for 2007-08. (return to source paragraph)
    
    
16. National Aboriginal Health Organization, undated, RHS 2002/03 Children's Survey Highlights: The Good, The Bad and The Ugly, mimeo. Statistic Canada's National Population Health Survey has been following the same individuals every two years since 1994-1995. It looks at how lifestyle, family and economic factors affect the population's health over time. (return to source paragraph)
    
    
17. Stan Auerbach, 2007, Special Needs Students in First Nations Schools: Inclusion in School Based Special Education Programs, FNESC Education Evaluation Report, p.34. (return to source paragraph)
    
    
18. Den Heyer, Isabel, Fred Wein et al, 2001, Mi'kmaq Students with Special Education Needs in Nova Scotia. (return to source paragraph)
    
    
19. George Fulford, 2007, Sharing Our Success: More Case Studies in Aboriginal Schooling, Kelowna: Society for the Advancement of Excellence in Education. (return to source paragraph)
    
    
20. Note that findings are based on self-identifying Aboriginal students. Government of British Columbia, Ministry of Education, 2006, Students with Special Needs – How Are We Doing? Province – Public Schools Only. See also Government of British Columbia, Ministry of Education, 2006, Aboriginal Report – 2005/06 How Are We Doing? Province – Public Schools Only. Government of British Columbia, Ministry of Education, 2001, Additional Considerations Arising from the Report on the Over- Representation of Aboriginal Students Reported With Behaviour Disorders; and McBride Management Limited, 2001, Over-representation of Aboriginal Students Reported with Behaviour Disorders, A Report to the Ministry of Education – B.C. Aboriginal Education Branch, April 2001. (return to source paragraph)
    
    
21. Stewart Clatworthy, 2006, Projections of Registered Indian and Aboriginal Populations, Households and Families. (return to source paragraph)
    
    
22. See, among others, CECASN, A Northern Light, (2005: 6), cited in Centre of Excellence for Children and Adolescents with Special Needs / Canadian Policy Research Networks Inc., 2006, Meeting the Needs of Children and Adolescents with Special Needs in Rural and Northern Canada: Summary Report of a Roundtable for Canadian Policy-Makers. (return to source paragraph)
    
    
23. SEP Annual Report Summary 2003-04, Version 4.0, December 12, 2004; Special Education Program Annual Report 2004-05, mimeo; and Special Education Program Annual Report 2005-06 (All Schools, excluding Alberta and Manitoba FNRMOs), mimeo, (All Annual Report Data was provided by Education Branch to AES; the 2005-06 aggregation was provided to AES by the Branch on September 19, 2007). (return to source paragraph)
    
    
24. In B.C, "School age student FTE" is calculated based on 8 courses being one FTE (full time equivalent.) Each full course (four credits) is funded as 0.125 of an FTE, up to 8 courses. A base minimum of 0.5 FTE is provided for secondary school age (non-graduated) students  . (return to source paragraph)
    
    
25. INAC, Education Branch, Amendment, December 2007. (return to source paragraph)
    
    
26. See, for example, Elizabeth Noble and Alan Bowd, 2005 Definition and Identification of Children's Attention and Behaviour Difficulties with a Focus on Northern Youth, Centre of Excellence for Children and Adolescents with Special Needs (with the support of the Public Health Agency of Canada, Lakehead University, Government of Nunavut, Memorial University, Mount St. Vincent, Nova Scotia and the University of Northern British Columbia). (return to source paragraph)
    
    
27. Arthur J. More, Ph.D., More & Associates, Consultants, Inc., 2006, Coordinated Student Assessments: Final Report, 2005/6, revised July 2006, A Report to the First Nations Schools Association and the First Nations Steering Committee. (return to source paragraph)
    
    
28. INAC, Education Branch, Amendment, December 2007. (return to source paragraph)
    
    
29. Ibid. (return to source paragraph)
    
    
30. This table does not include allocation and expenditure data for 2002-03 (According to Department Officials, expenditures were not tracked due to the late release of funds prior to the new fiscal year). (return to source paragraph)
    
    
31. Prairie Research Associates, 2003, Evaluation of the Official Languages in Education Program, prepared for the Department of Canadian Heritage (June 25, 2003). (return to source paragraph)
    
    
32. Council of Ministers of Education, Canada, (2004), Quality Education for All Young People: Challenges, Trends and Priorities: Report of Canada, Prepared in response to the International Survey in Preparation for the Forty-Seventh Session of the International Conference on Education, Geneva, September 8-11, 2004,  . (return to source paragraph)
    
    
33. According to First Nation Authorities, the final report of the pilot recommended that the rubric be implemented at the national level for the Special Education Program. (return to source paragraph)
    
    
34. SEP's Terms and Conditions were approved prior to the completion of INAC's 2005 Education Action Plan, and thus make no reference to the National Educational Dashboard (NED) which followed upon this Plan. NED pulls together data from three sources: 1) Nominal Roll System; 2) National Post- Secondary Education System; and, 3) Indian Registry System, and allows the display and reporting of aggregate level data of a predefined set of education indicators. Further, an Education Tracing System allows for longitudinal analysis of a cohort of students by tracking their progress through the education system year after year. (return to source paragraph)
    
    
35. Based on INAC, 2000, Corporate Information Management Directorate, Issue No. 4 June 2000, 9. (return to source paragraph)
    
    
36. A non-registered student who is ordinarily resident on reserve lands that are leased is not eligible for funding. (return to source paragraph)
    
    
37. This discussion is based on a review of one of the three different reporting forms used by FNRMOs to report on SEP activities in 2005-06. (return to source paragraph)
    
    
38. A 2005 INAC Information Management Branch report, Transforming INAC's Data Collection and Reporting, said data collection was not driven by performance measurement objectives and FNRMO and school reports were not timely, complete, or efficient, and recommended that "…unless the data is going to be used and shared to improve the program, it should not be collected. (return to source paragraph)