Follow-Up Audit of the Education Information System

Education Programs

The Department's Education Programs, managed primarily within the Education and Social Development Programs and Partnerships (ESDPP) Sector, and delivered by the Regional Operations Sector, are designed to support First Nation and Inuit students in achieving educational outcomes comparable to those of other Canadians. The goal of the Department's Education Programs is to provide eligible First Nation and Inuit students with quality education and the opportunity to acquire skills that will prepare them to enter the labor market and participate in Canada's economy.

Education Information System (EIS)

The EIS is a computer database that manages the information relating to the Department's Education Programs. The system was implemented to modernize and replace paper-based processes for reporting and to streamline data collection practices. The system provides valuable additional data in a central location, which is intended to aid the Department and First Nations in strengthening accountabilities in the delivery of First Nation education by tracking performance, the movement of students, their academic achievement, and measuring success of Education Programs. The resulting information will allow the Department and First Nations to gain a better understanding of the performance of students and schools, address concerns and make enhancements where necessary, to support improved educational outcomes, in a timely manner. First Nations adoption of EIS and their use of the data will improve as the benefits of using the system are demonstrated, as capacity improves and user training is provided. All First Nations are currently using EIS in some capacity. Data entered into the EIS can be done through a Portal which is a seamless departmental gateway to a variety of systems, reporting requirements and tools including the Education Information System, the Education Reports and Analysis Solution (ERAS), and other departmental reporting systems.

EIS was designed for First Nations to log in and submit education-related information to fulfill reporting requirements and run reports for their own monitoring purposes. The rate of adoption of the system is therefore a key indicator of EIS' success.

During the audit, it was noted that a limited number of First Nations were able to use the full functionality of EIS (i.e. leveraging the Portal to enter, review and submit education-related information for proposal and reporting requirements). EIS is supported by the ERAS, which is the reporting tool used by regional and recipient users to access general and customized reports to inform timely decision-making.

5.1.1 Transition Plan

Given that EIS was in its transition phase during the audit period, moving from a project to maintenance, it was communicated to the audit team that a formal Transition Plan had been established. Formal transition plans typically set out the key objectives and targets to be achieved by the end of the transition phase. In setting these objectives and targets, the project team would be defining the requirements to move EIS to maintenance. Establishing these objectives and targets would allow the project team to assess its progress throughout the year and enable management to determine whether the system has achieved its maintenance as scheduled. In addition to transition phase objectives and targets, other key elements of a formal transition plan include the associated governance structure and framework, roles and responsibilities of stakeholders, knowledge transfer and training activities, key IT processes and activities, as well as funding requirements for the transition period. Many of these elements were observed in different EIS-AU documents. However, the audit noted that a formally documented Transition Plan that outlined the objectives and targets, detailed activities and associated funding specifically for the transition phase was not established. As a result, it was challenging to determine how the project team defined the system's maintenance and what was required for the system and associated processes to achieve this result.

In lieu of formally defined objectives and targets for the transition phase and the subsequent maintenance, the audit team looked to the original project goals, objectives and critical success factors established at the beginning of the EIS project in 2010-2011. In doing so, the audit team assessed EIS against the four established critical success factors, which underpinned the project's goals, objectives and business outcomes. These factors were to be met by the end of the original EIS project completion (revised date of March 2014) in order to be assessed as providing the necessary benefits relative to the investments made. The critical success factors are highlighted in the table below along with an assessment against these factors:

As demonstrated in the table above, while the audit team noted that elements of these critical success factors have been attained, not all of them have been fully achieved nor are they expected to be achieved by the end of the transition phase. As a result, the overall EIS project objectives and goals have not been fully realized.

In gaining an understanding as to how and why EIS has not fully achieved its critical success factors, it was noted that the objectives for EIS and EIS-NG were not fully achieved at the time of project close, and that outstanding items were moved forward to the subsequent phase. The impact of this resulted in the transition phase primarily focusing on the implementation of key functionality to improve the system and achieve the outstanding objectives rather than the transition of the system to maintenance (i.e. ongoing system upkeep and transferring knowledge and responsibilities from contractors to staff). Secondly, it was noted that certain constraints (i.e. the Government of Canada requirement that all systems be built on an Internet Explorer platform) resulted in compatibility issues for many recipients, affecting the attainment of the system's critical success factors. As such, it was identified that over the five-year course of the system's development and implementation, several internal and external constraints affected the achievement of the original project goals.

In recognizing these constraints, the audit noted that a re-baseline exercise to re-establish system goals, objectives and critical success factors for EIS had not been undertaken to more closely align them with the current business, system and project realities. This would have allowed the team to define maintenance for EIS that reflects current constraints and position them to develop a transition plan designed to achieve maintenance.

5.1.2 Funding

Funding of $27.3M was allocated for the development and implementation of the EIS at the onset of the project with an additional $450K for each subsequent year for ongoing maintenance costs (effective 2013-2014). During the project, and for each additional phase, the original EIS budget was supplemented with additional funding for ongoing support and other functionality identified by the project team. In total, at the time of the audit, it was estimated that $56.2M was to have been provided for the EIS system and its upkeep by March 2016. The table below depicts the estimated approved funding allocated for each phase of EIS:

The audit team noted that a report called the Ongoing Support Activities, Organization, and Costs Report, isa key EIS-AU document that outlined the funding required to support the transition activities in 2015-2016. Supporting this funding amount was a list of high-level activities that the funding would be allocated towards, including ongoing support to regional and First Nations users, transition activities from consultants to staff, and implementing system functionality to improve EIS. Despite the articulation of these activities, no evidence was found of reporting or monitoring activities having taken place to track these funds against the activities they were expected to be used for.

Based on the lack of reporting in relation to key activities required to achieve maintenance, it is difficult to assess whether EIS is on track to achieve maintenance within the allocated budget. Some concerns were raised in oversight committee meetings regarding the funding remaining for 2015-2016 as well as funding allocated for 2016-2017 and whether it was sufficient to carry out ongoing support activities. Given the fact that formal transition phase targets were not established and only high level activities were defined with limited monitoring of these activities against budget, it is not clear how much funding is required for EIS to achieve maintenance and whether this is within the existing funding allocation provided for EIS.

Recommendation:

3. The Assistant Deputy Minister of the Education, Social Development Programs and Partnerships sector, in collaboration with the Senior Assistant Deputy Minister of Regional Operations should ensure that a sufficient and reasonable training plan is developed to support the provision of EIS training to First Nations. This plan should be supported by the resources and tools necessary to enable regional representatives to carry out their training responsibilities.

5.3.1 System adoption

The EIS system, in particular the Portal, was designed for First Nations to log in and submit education-related information to fulfill reporting requirements and run reports for their own monitoring purposes. The rate of adoption of the system is therefore a key indicator of EIS' success.

During the audit, it was noted that a limited number of First Nations were able to use the full functionality of EIS (i.e. leveraging the Portal to enter, review and submit education-related information for proposal and reporting requirements). A key reason for low adoption is system compatibility issues relative to the platform that EIS was built upon, Internet Explorer, as per the Government of Canada's requirement. EIS, the Portal and the ERAS reporting tool, have been built on and can only be accessible through the use of Internet Explorer. However, many First Nation recipients do not use Internet Explorer and as a result, these recipients cannot access EIS.

A solution that was provided to First Nations is the use of encrypted USB keys with attached smart PDFs. This has been established as an alternative solution for recipients who are in remote locations and have limited internet accessibility. This is a process whereby regions provide encrypted USB keys with attached smart PDFs to First Nation recipients who then complete the smart PDFs and save the information to the encrypted USB. The First Nation then mail the encrypted USB key back to the regional office and a regional representative uploads the PDF to EIS. The regional representative then completes the data validation process relative to the information provided by the First Nation.

The audit noted some performance issues with uploading PDFs to EIS. As further discussed in Section 5.3.2, this process can be time-consuming and system timeouts have been experienced due to the large size of some PDF files. As the regional representatives complete the data validation on behalf of the First Nation, warnings and error messages are sometimes noted which requires the regional representative to communicate with the recipients until the errors are resolved. This has resulted in an increase in workload for some regional representatives as well as delays in the overall process.

The compatibility issues experienced by many First Nations also affected their access to reporting capabilities within EIS and ERAS, which does not allow them to leverage existing reports that could help them make better informed, more timely decisions relative to education. As a key stakeholder for the system, it is important to have a relevant long-term vision on how First Nations are expected to interact and leverage the capabilities within the system.

5.3.2 System Performance

The audit included an assessment of the measures in place to enable adequate system performance of the EIS. As noted in Section 5.3.1, a limited number of First Nations are currently using the full functionality of EIS given compatibility and accessibility issues. The challenges with the performance of the system have affected many First Nations' ability to fully adopt the system. As a result, there was an increase in workload for some regional representatives as parallel tools and processes were maintained within the regions.

The use of electronic submission documents in the form of the smart PDFs sometimes results in large files being uploaded into EIS, particularly for Nominal Roll. The upload of these large PDF documents often resulted in system timeouts and errors. Large PDF files put significant strain on the system and can result in system failures.

System capacity is another factor that affected the performance of the EIS. If there is not sufficient capacity to maintain several users in EIS at the same time, the processing time within the system increases and the system can fail. The audit identified that the responsibility for the management of the infrastructure and environment lies with Shared Services Canada (SSC) and therefore, the Department must rely on SSC to track and monitor system capacity usage and logs. The information from the system capacity reports and logs is shared with IMB; however, the audit noted no evidence of formal, periodic monitoring of the capacity reports or logs. The audit also noted two instances of load testing that were performed over the capacity of EIS by IMB, and it was revealed that the system would consistently time-out when there was upwards of 42 users using the system simultaneously. If left unresolved, this will become a growing concern as the Department sees an increase in First Nations using EIS to submit their education-related information.

The audit also revealed that there were coding errors and defects that result from each new release of the system. In the EIS-AU phase alone, from April 2015 to March 2016, five releases are expected to be completed. Regional representatives noted that the new releases often resulted in unresolved coding errors and defects that further impeded the performance and hindered the successful use and adoption of the system.

As a result of these performance issues, regions continue to use parallel processes and tools (e.g. the use of excel spreadsheets and Crystal reports) to track and report on the collected education data from the recipients. This has created additional workload and duplication of effort for some regional staff, as they are responsible for ensuring the successful submission of education program data (in consultation with the recipients to resolve any errors and warnings resulting from the EIS data validation process), as well as maintaining separate systems to track the data already within EIS. However, overall, workload is still less than in previous years, as proposals are verified automatically and there are automated compliance reports that help reduce previous levels of work.

5.3.3 Reporting

The audit assessed the timeliness and reliability of the reports from EIS and ERAS and the effectiveness of the reporting functionality of these systems in supporting performance measurement and timely decision-making.

The audit found that the EIS reporting tool, ERAS, has been developed with a number of general and custom reports available for both the Elementary/Secondary and PSE programs. It was noted that due to some data integrity issues with PSE data, it was not transferred to ERAS to enable the reporting capabilities for this program. As such, at the time of the audit, the Department was not yet able to generate reports, statistics or indicators for PSE from ERAS. Nominal Roll data is available in ERAS as of the year 2000, and as such, reports related to Nominal Roll are available to support departmental performance reporting for the Elementary/Secondary education programs. Education Branch representatives confirmed that PSE data was being reviewed and cleansed and would be available for reporting and analysis purposes in fiscal year 2016-2017.

Interviews with regional staff revealed that although they were provided training and access to use ERAS, the reporting tool was not yet being used in some regions. In addition to the lack of reports available for PSE , the Nominal Roll reports could only be generated from ERAS once all of the data from each region was been submitted for the fiscal year. The regions require real-time reporting data for decision-making purposes, and as a result, parallel tools and processes were still being leveraged by the regions for reporting and management decision making purposes. Also, given the accessibility challenges with EIS for First Nations, First Nations were not yet able to leverage the current reporting capabilities of the system.

5.4.1 Data Integrity

A key function of EIS is the ability to validate the data against pre-defined business rules. These validation rules improve the integrity of the data within EIS, which in turn, allows for improved reporting and decision making. The audit found that adequate validation controls exist during the upload of data to EIS and in the interfaces with other systems to ensure the integrity of the data. When data is directly input into the system through the Portal, reliance is placed on the business validation rules built within EIS to ensure the accuracy and completeness of the data entered. In cases where data cannot be input directly into the Portal, the integrity of the data prior to the upload into EIS is ensured through the use of:

• Smart PDFs, which contain mandatory field requirements, business rules and use dropdown menus with pre-defined values; and
• XML files from a School Information System that have been tested against a test schema provided by the Department prior to submission of data.

Within EIS, there are several business validation rules that have been defined, documented and implemented by module for reports and proposals. Depending on the nature of the business rule, errors and warnings will result. Errors must be resolved before the data is submitted; however, the audit noted that the warnings will not prevent submission of the data. As an example, a key validation check prevents users from entering duplicate students into the system. This validation is performed against the student enrolment field, i.e. duplicate student enrolments within the same Nominal Roll for a given fiscal year are identified during the system validation and generate an error. In addition to the validation check, a duplicate report can also be generated to compare students against multiple Nominal Roll reports across regions.

Another example of the validation rules within EIS is the matching of the IRS number within EIS. Validation rules within EIS match the student information against the data in IRS. This includes a validation check against the IRS number, last name, date of birth and gender. Through the audit procedures, these validation rules were confirmed to be in place to demonstrate the existence of controls relative to the maintenance of data integrity in EIS and through the interfaces with other systems.

5.4.2 Data Security and Privacy

In accordance with the TB Operational Security Standard: Management of Information Technology Security (MITS), it is expected that systems are certified and accredited before they are approved for operation. Additionally, it has been established that the accreditation should be reviewed on a periodic basis in cases where the system has changed significantly or if there have been changes in the risk environment. The certification and accreditation verifies whether the established security requirements are met and that the controls and processes in place are operating effectively.

The original certification and accreditation of EIS in December 2012 resulted in EIS not meeting the mandatory security requirements, and was not approved for operational use.

A business decision was made to continue with the implementation of the system into production while working to mitigate the security concerns. In April 2014, EIS underwent another certification and accreditation process and the system was granted Interim Authority to Process. Interim Authority was granted as there were some outstanding issues surrounding deficiencies found during the architecture and code reviews and TRA. This Interim Authority expired in September 2014 and during the audit period there was no certification and accreditation for EIS despite the fact that that system continues to operate in production and ongoing changes were still being made to the system.

Given the sensitive nature of the data entered into EIS, the audit examined the segregation of production data from the development and test environments. Through examination of system screens and inquiry with representatives from HQ, it was determined that EIS is segregated into several environments (i.e. production, development, testing); however, production data is copied to these other environments on a periodic basis. This can be a concern as users in development and test environments without access to production, may have unnecessary access to the sensitive/private production data (such as student names, IRS number, date of birth and financial data). This access may not be in line with their job responsibilities or align to their current level of access to data in the production environment.