Audit of Post-Secondary Education Programs

The Post-Secondary Student Support Program (PSSSP) and the University and College Entrance Preparation Program (UCEP)

AANDC provides support to eligible First Nations and Inuit students for PSE through PSSSP and UCEP Program. The objectives of these programs are listed below.

• The objective of the PSSSP is to improve the employability of First Nations and eligible Inuit students by providing them with funding to access education and skills development opportunities at the post-secondary level.
• The objective of the UCEP Program is to provide financial support to First Nations and eligible Inuit students who are enrolled in university and college entrance preparation programs offered in Canadian post-secondary institutions, and to enable them to attain the academic level required for entrance into degree and diploma credit programs.

PSSSP and UCEP Program are the primary components of the Department's PSE programming. Spending in 2013/2014 for these programs was approximately $314 million or approximately 93% of total available PSE funding.

Funding flows from AANDC Headquarters (HQ) to the regional offices, who then allocate funding to First Nations' organizations, such as band councils, tribal councils, or representative organizations. Eligible Inuit students are funded through designated aboriginal organizations or through some AANDC regional offices. These funding recipients then administer the funds to eligible students within their respective communities. As such, the Department has limited ability to influence how funding is ultimately allocated to students, which provides a challenge to the Department's ability to directly influence Program outcomes.

The Post-Secondary Partnerships Program (PSPP)

AANDC provides support directly to post-secondary institutions through the PSPP. The objective of the PSPP is to assist Canadian post-secondary institutions in the design and delivery of college and university level courses that respond to the educational needs of First Nations and Inuit students. It is expected that the program will increase the availability of post-secondary courses and programs targeted to First Nations and Inuit students in fields of high labour market demand, either locally, regionally or nationally, and in the fields of Aboriginal governance and business development.

Planned spending in 2014/2015 for the PSPP was approximately $22 million, with $7 million allocated to support the operations of First Nations University of Canada, which is headquartered in Regina, Saskatchewan. The remaining $15 million is disbursed to fund approved projects from eligible post-secondary institutions after a proposal intake and review process. In 2014/2015, a total of 87 projects from 35 post-secondary institutions were approved.

5.1.1 Policies, Procedures and Tools

As with any AANDC program, there was an expectation that Program and Regional Management develop and implement a regime of processes, procedures and supporting tools to promote, and monitor the achievement of, Program objectives as well as ensure compliance with requirements. This regime should reflect the Department's expectations for the management and control of programs, as well as reflecting the context and risk profile of the PSSSP/UCEP. As a minimum, the audit expected to confirm the existence of well-designed and effective funding agreements, performance measurement frameworks, monitoring and compliance regimes, as well as sufficient guidance, tools and training to support both the regional offices and program recipients in fulfilling their respective roles. Except for monitoring capabilities and practices, which are described in Section 5.1.4 of this report, this Section provides the results of Audit's assessment of policies, programs and tools as they apply to PSSSP/UCEP.

The audit procedures identified that funding agreements and national program guidelines are key foundational elements supporting PSSSP/UCEP objectives. Examination of a sample of funding agreements revealed that they fully reflect AANDC's standard templates. While, as noted in Section 1, the Department has limited ability to influence how funding is ultimately allocated to students, the audit confirmed that funding agreements included Terms and Conditions (T&Cs) which promoted the use of program funds for intended purposes. Examples of T&C s that promote the appropriate use of PSSSP/UCEP Program funding include those related to:

• All funding agreements state that the recipient must adhere to the terms and conditions of the program as set out in the NPG (see below for further details).
• Unlike block funding agreements, recipients in a fixed funding agreement are not allowed to shift PSE funds away from PSSSP/UCEP without an unexpended funds plan. At the end of their current agreement, any surpluses may be recovered from the recipient. These features are aimed at encouraging the use of funds for intended purposes.

As noted above, all funding agreements include a requirement that recipients adhere with specific T&C s that are detailed within the NPG. The program guidelines were developed by the Program to articulate a common set of expectations applied to recipients including the expectation of PSSSP/UCEP related reporting. Examination of the current national program guidelines indicated it provides detailed guidance concerning selection priority criteria for funding decisions, the appeals process, and the requirement that recipients establish publicly available Local Operating Guidelines (LOG) that align to the requirements in the NPG. The funding agreements also require that recipients provide specific financial and performance information via annual Audited Financial Statements (AFS) and the departmentally provided Data Collection Instrument (DCI) respectively on an annual basis. Audit work conducted on-site at a sample of 4 Regional offices further indicated that some Regions have developed additional tools to assist recipients in complying with the national program guidelines. For example, the Yukon Region shared a locally developed budget allocation template and its Regional Operating Guide while BC Region had developed a DVD aimed at helping First Nations address program reporting requirements.

The audit also identified that an updated Performance Management Strategy (PMS) for Education had been developed in early 2014/15.  This PMS included 6 performance indicators for PSE, each of which link to outcomes in the Education Branch's Program Logic Model.  Due to changes to the data collection processes, further described in Section 5.1.3, there have been some delays in the operationalizing of performance reporting in support of the PMS.

Finally, the audit revealed evidence that Headquarters (HQ) Program Management had taken initiatives to improve and update the tools and guidance provided to Regions and Recipients. For example, examination of the current NPG and prior program guidelines revealed evidence of updates that serve to better define eligible expenditures under PSSSP/UCEP. We also identified a recent ESDPP sector initiative to develop a Draft Program Compliance Directive and Draft Program Recipient Compliance Review Handbook, which includes guiding principles, roles and responsibilities, procedures for conducting risk assessments and developing risk-based compliance plans, and procedures for conducting compliance reviews. These tools have not been implemented for PSE yet, and are designed to be used across ESDPP programming.

5.1.2 Approach to Program Funding

Except for a relatively small volume of grants provided by certain Regional offices directly to eligible students, recipients are responsible to fund individual students in accordance with the national program guidelines. Based on a funding mechanism, each region receives PSE funding from HQ which they, in turn, distribute to recipients. As noted earlier, regional offices are afforded a degree of flexibility in the administration and delivery of programs. Nonetheless, the audit expected to identify funding mechanisms and funding allocation methodologies that promote consistent implementation of the PSSSP/UCEP Programs across Canada and equitable access to funding by eligible students in different regions. Moreover, there was an expectation for effective mechanisms to monitor, assess and address the alignment of Program objectives with Program resource capacity, such as needs versus funding level analysis, as a means to help align funding practices with funding needs.

The program has three types of funding: block, fixed and grants. Block funding for 2013/14 was approximately $132M, fixed funding was approximately $179M, and grant funding was approximately $1M. For block funding arrangements, the Department applies a formula to apportion an amount to PSE programming out of the total block funding allocation (as set out in the block agreements) for each region. The Region then allocates to individual recipients based on their respective block agreements. PSE recipients that are funded under block arrangements are allowed to reallocate funds to other programs provided they can demonstrate progress towards achievement of PSE's Program objectives.

For fixed funding arrangements, Program funding is based on historical funding patterns subject to a nominal annual increase of approximately 2%. The Department may allocate the 2% to mandatory expenditures in other educational programs. This is consistent with the approach identified in the 2009 internal audit. This approach has not been informed by any review or analysis to ensure that HQ allocations to the regions promote the objective of equitable access to funding across all regions. As a result, there is an increased risk that achievement of this objective will be diminished over time.

The 2009 internal audit also included a recommendation for a review of the sufficiency and appropriateness of recipient funding requirements. To date there has not been a formal needs assessment to assess the sufficiency of funding to recipients relative to program objectives. There are some practical challenges associated with conducting an effective needs assessment, such as the ability to identify potential students living off-reserve. Moreover, the issue of sufficiency of funding at the student level is complicated, and made more sensitive, by a number of contextual and historical factors as described in the Background section of this Report. While defining the sufficiency of funding is an important consideration for Program management, this audit did not include a detailed assessment of such an analysis. Irrespective of funding amounts, the nominal increases to annual funding are unlikely to match the well-publicized increases in the costs of PSE across Canada and, as such, there is a risk that delivery of program objectives will be eroded over time. In addition, the variability of demographic profiles across the country can impact where funding needs are most acute.

Also identified in the 2009 Audit was the fact that funding allocation methodologies were not consistent across all regions. This audit confirmed that regions continue to exercise some flexibility in how a portion of its funding is allocated. This flexibility allows individual regions to take initiative by implementing a funding allocation approach that they consider to be in the best interests of both the Program objectives and the Recipients in their region. For example, in one region visited the allocation methodology for Fixed funding incorporated some consideration of recipient funding needs. This was accomplished by redirecting some of the overall regional funding, that is above an initial allocation amount, into pools of funds from which additional funding may be accessed by recipients through a needs-based proposal process. Recipient needs are demonstrated through planned uses of funds and student waiting lists. This process was developed in collaboration with stakeholders that included recipient representatives. As evident from this example, small differences in funding methodologies are not inherently inappropriate. However, inconsistencies present a risk related to the fact that recipients in one region may not have the same access to additional funds as a similar recipient in another region. Moreover, the audit found that unique methodologies employed in some of the visited regions had not been reviewed or revised for some time.

Recommendation:

1. The Assistant Deputy Minister of ESDPP should review the appropriateness of how PSSSP/UCEP funding is allocated across regions to ensure it supports equitable access and, to the extent possible, promotes the alignment of funding practices with funding needs. Based on the outcomes of this review and overall Program objectives, the ADM of ESDPP should develop national guidance to support the consistency of funding allocation methodologies employed by regions while enabling regional differences when warranted.

5.1.3 Changes to the Data Collection Processes

The collection of relevant and accurate performance information is vital for measuring how well the PSE Program is achieving intended results and planning effectively for the future. As with many other AANDC programs, the efficient and effective collection of recipient data is an ongoing challenge for PSSSP/UCEP. Previous audits and evaluations identified the need for more rigorous collection of performance information in order to support management decision-making, planning, and monitoring of PSE programs. In 2012/13 new recipient reporting requirements for PSSSP/UCEP were introduced as part of a transition to the Department's Education Information System (EIS). The audit expected that these changes to the data collection processes would result in an improved ability to monitor PSSSP/UCEP recipients and otherwise serve to improve overall programming.

Funding agreement T&C s require that Program recipients provide two key information sources. The first source is the annual AFS which include schedules designed to provide financial details of the recipient's use of funding, including a schedule outlining the use of PSSSP/UCEP funding. During the audit, there was evidence that AFS were being monitored by regional personnel for compliance with limits for spending on program administration costs and eligible expenditures. Section 5.1.4 provides further details on the nature and effectiveness of AFS monitoring.

The other key source of information is the Program's current reporting tool, referred to as the Data Collection Instrument (DCI), which is a fillable forms that all recipients must complete annually.. The DCI facilitates collection of a variety of information from program recipients on funded students. The DCI template includes instructions for the recipient on how to complete the forms and interviews revealed that some regions have provided DCI-related training to recipients. Once completed, the Region is responsible to review the DCI and then enter/upload the data to the EIS. Audit procedures conducted in the four regional offices visited indicate that the DCIs are being reviewed to ensure they are complete and that any error reports (e.g. recipient entered or attempted to enter invalid information such as an unapproved post-secondary institutions) have been cleared. As described more fully in Section 5.1.4, the monitoring does not extend to assessing the quality of information entered into the DCI before it is uploaded to EIS.

The EIS aims to support improved accountability for education programs, inform changes to policy and program development, and improve service delivery. While EIS is being rolled out within Education Branch, it has not yet fully operational in respect of PSE programs, subject to completion of data uploading and compliance validation. There has been a backlog of data input to EIS at the regional level whereby 2012/13 data was only fully input in 2014/15 and Education Branch is currently being assessed for compliance with T&C s and NPG in preparation for reporting and analysis. Therefore, while PSE information has been collected from recipients and entered into EIS, this information is not yet leading to the generation of reports that can serve to support management in compliance monitoring, decision-making, accountability or performance measurement. Further, the lack of reporting has meant the Program has little to no ability to assess the quality of reported information. Interviews with Program management and EIS representatives indicate that the Program has not yet documented a clear plan for how and when EIS reporting will be leveraged for PSE. The information gap created by the lack of PSE reporting , as EIS is rolled out to PSE programs, has significantly impaired Management's ability to support and report on the achievement of program performance objectives and monitor the extent to which recipients are in compliance with Program requirements.

Recommendation:

2. The Assistant Deputy Minister of ESDPP should develop a clear and formal strategy and plan as to how and when EIS reporting capabilities will be leveraged in support of the effective management and control of PSE programs (i.e. what reports, how often, to whom, etc.) as well as how training will support improved accuracy and consistency of data collection.

5.1.4 Approach to Monitoring

The audit included an assessment of the extent to which monitoring and oversight activities were conducted to promote compliance with program requirements and achievement of program objectives. Further, the audit assessed those mechanisms in place to monitor, assess and address the alignment of Program objectives with Program resource capacity. To this end, the audit procedures included interviews and sample testing in four regional offices to identify and assess processes for monitoring recipient's compliance with funding agreement T&C s including the national program guidelines. We also conducted independent assessments to identify the extent to which compliance objectives were achieved. Finally, we identified and assessed the processes in place at HQ to support alignment of resources and otherwise support effective monitoring of compliance and program objectives.

While there was some variance in the depth of monitoring, audit procedures conducted in the four (4) regions visited indicated that Regional personnel were conducting limited in-office monitoring, comprised of the review of Audited Financial Statements (AFS) and the review of the DCIs. The AFS review was conducted for purposes of examining program expenditures to determine compliance with the requirement that administration costs not exceed 10%. The DCIs were found to be tracked and reviewed for purposes of confirming receipt and completeness of the templates.

In none of the regions visited was there evidence of comprehensive compliance monitoring against all funding agreement T&C s or the NPG. Specifically, the breadth and depth of monitoring or compliance activities undertaken would not appear sufficient to routinely identify potential compliance issues or to fully promote the achievement of Program objectives. For those limited compliance activities conducted, follow-up actions taken with recipients were also found to vary across the regions visited and such practices were not formally established. Even when monitoring activities were identified, audit procedures revealed that instances of non-compliance were not always followed-up or addressed. For requirements that are not monitored at the regional level, such as the requirement for recipients to develop and publicize LOGs, high rates of non-compliance were identified. Of the audit sample of 45 recipients, 15 LOGs were found to be available online. These 15 were reviewed for compliance to the program guidelines, and 14 contained the basic requirements set out in the NPGs in terms of having selection criteria and an appeals process. However none of the LOGs met all the required elements of an "appeals process" as set out by the NPG. For greater certainty, there was no evidence that regions have taken a risk-based approach to monitoring PSE recipients. While the General Assessment Process is applied to all program recipients as a means to assess their capacity, this process does not provide sufficient insight to inform a risk-based approach to monitoring. Without a comprehensive, risk-based, approach to monitoring the likelihood of not achieving program objectives is increased as is the likelihood that recipients will not be in compliance with funding agreements T&Cs.

Based on interviews with regional and HQ personnel, there was a common understanding that resource levels at the regional level are a factor in the breadth and depth of PSE-related monitoring. For example, information obtained from a region not visited during this audit provided details of a regional PSE compliance program whereby compliance with PSSSP/UCEP T&C s and program guidelines was recently assessed on a student-by-student basis. Interviews with that region revealed evidence of compliance issues which, in some instances, involved the recovery of funds.

Notwithstanding the concerns raised above, the audit identified that ESDPP is currently in process of developing Sector-wide compliance tools including a new Program Compliance Directive and Program Recipient Compliance Review Handbook which, when rolled out to PSE programs, will address the conduct of compliance reviews for PSE. These documents contain draft PSE-specific compliance checklists developed by Education Branch. The audit also identified a parallel initiative led by Regional Operations Sector to address overall resource levels for compliance. This initiative is seeking to promote consistent and efficient compliance activities that are aligned with resource capacities through strategies that include developing common structures and resource classifications While these are positive developments, the timeframe for implementation of the new Sector-wide compliance regime to PSE programs will not likely occur until 2016-17. A sound implementation plan will be necessary to assure the effectiveness/sustainability of this new compliance regime. Development of such a plan will necessarily involve engagement with regions on such matters as reporting, tools, and resourcing. In the interim, a minimum standard for compliance should be established, based on recipient risk, in order to help address the compliance gaps identified and to mitigate the risk that funds aren't being expended for intended purposes.

Recommendation:

3. The Assistant Deputy Minister of ESDPP should establish an implementation plan and timeline for implementing ESDPP's new compliance regime within PSE. In the interim, develop a risk-based approach to monitoring whereby a minimum standard for compliance monitoring and follow-up across all regions is established and sustained.

5.1.5 Management of PSSSP and UCEP Grant funding

Three (3) regional offices administer approximately $1M grant funding: Yukon ($400k), Alberta ($500k), and Saskatchewan ($60k). Yukon and Alberta administer the program on behalf of those First Nations which have made arrangements with the regional office to administer the program on their behalf. Saskatchewan administers grants to Inuit students from that have resided outside Nunavut or the Northwest Territories for 12 consecutive months and are no longer eligible for PSE funding through their respective territorial governments.

The audit site visit to Yukon revealed that the region had established a Regional Operating Guide to administer the program. This Guide functions in the same manner as a LOG developed by other program recipients. The region has also established appropriate practices and protocols for applicant student intake, documentation requirements, and funding calculations. They had also developed a compliance and assessment checklist to determine whether the student is eligible to receive funding. Audit testing completed a sample of grant files indicated that controls were sufficiently effective.

Saskatchewan was found to have funded only 4 students in 2013/14. While the audit noted that granting processes were found to ad hoc, audit testing completed on a sample of grant files indicated that controls were sufficiently effective.

5.2.1 Program Objectives

The audit examined the process undertaken by program management to define PSPP objectives as part of its redesign. The audit team expected that a program concept and objectives for the PSPP would have been established that addresses a need as determined through consultations or other means, and that are aligned with the Departmental mandate and priorities.

The audit found that departmental senior management requested that the former ISSP program be reviewed and renewed to ensure that the management and delivery of the program was aligned to its original intent and objective, and to government and departmental priorities. As such, the program was altered to include a more rigorous and transparent proposal review process to select projects to fund (discussed in greater detail in section 5.2.3 below), and on funding projects that would contribute to meeting labour market needs (either locally or nationally) and lead to higher employment among Aboriginal youth. To assess whether projects were addressing a local or national need, the program relied on the Employment and Social Development Canada's (ESDC's) labour market needs list and the knowledge of regional staff. While these sources may provide some useful insight into assessing need, without undertaking a formal needs assessment that included stakeholder input, the extent to which the program is considering the greatest needs of First Nations' communities and students is not clear.

Additionally, the audit team was informed that program redesign was intended to introduce a greater emphasis on establishing partnerships, where proposed projects that included additional funding partners would receive preference over those without. The name of the program was also changed to include the word "partnerships" to stress this focus. The intent was that this would allow the Department, and the funding partners, to lever off the other's contribution to achieve results. Program management has embedded these new areas of focus (i.e. labour market needs and partnerships) into the rating scoring criteria used by staff to assess proposals.

The audit also reviewed Education Branch's Performance Measurement Strategy (PMS) to determine the expected results for the PSPP and defined indicators of program success. The audit team found that the PMS does not include any PSPP-specific outcomes and indicators.

Overall, the audit found that the primary objectives of the PSPP were not clearly reflected or reinforced through the program Performance Measurement Strategy and the proposal review process. While the 2014/15 Departmental Report on Plans and Priorities describes the importance of stronger relationships to labour market needs, this factor comprises only 20% of the rated scoring applied to proposals. Further, labour market alignment is not identified as a mandatory requirement. This effectively diminishes the importance of this objective as projects could still be approved without scoring well in the area identified as most closely linked to Departmental objectives. Furthermore, it is unclear how program performance will be assessed or what constitutes program success without clear program outcomes and indicators. This lack of clarity in program objectives and outcomes presents further challenges in identifying the alignment of the program with the Departmental mandate and priorities.

5.2.2 Policies, Procedures, and Tools

The audit also examined the policies, procedures, and tools that are in place to assist staff in program delivery. The audit team expected to see updated program documentation that sets out program standards, roles and responsibilities of staff, and expectations of recipients.

The audit found that program management has updated the national program guidelines to reflect the redesign of the PSPP, and has provided it to regional staff. The updated program guidelines clearly set out the program standards in terms of eligible activities, target recipients, eligible expenditures, and other key characteristics for the program. The program guidelines also define the proposal assessment criteria and weighting for each criterion.

Program management has also developed a National Selection Committee (NSC) Terms of Reference, and has provided it to regional staff. The NSC Terms of Reference clearly sets out the roles and responsibilities and steps involved in the proposal assessment process.

Expectations of PSPP recipients are set out in their funding agreement, which includes undertaking project activities in accordance with their approved proposal and the national program guidelines. It also requires recipients to submit an annual PSPP report using the departmental PSPP DCI template.

While the funding agreements set out recipient reporting expectations, the audit found that the DCI template and its data elements have not been updated after the redesign of PSPP. As such, some data elements within the DCI template do not appear to be relevant to the redesigned program. For example, the categories of expenses within the DCI include expenses for elder/guest speakers which no longer appear as eligible expenses under the PSPP program guidelines.. Furthermore, as noted earlier, the program has not yet established PSPP-specific indicators, so it is unclear whether information collected from submitted DCIs will enable an assessment of program performance.

5.2.3 Proposal Review Process

A key component of the redesign of the PSPP was a revised proposal review process. Prior to 2013/2014, the regions allocated ISSP funds to recipients. Program management has since moved this process to a national-level through the establishment of the NSC, which still includes both HQ and regional involvement. The NSC is responsible for the review and approval of PSPP proposals. In 2014/2015 the NSC members included:

• The Assistant Deputy Minister of the ESDPP Sector (Co-Chair);
• The Senior Assistant Deputy Minister of the Regional Operations Sector (Co-Chair);
• The Director General of the Education Branch;
• The Director of Education Programs; and
• The 10 Regional Directors General (RDGs).

The proposal review process involved the regions submitting a call for proposals to potential applicants within its jurisdiction, and then receiving proposals from interested applicants. An initial review was then conducted of each proposal by the region that involved an assessment of eligibility and a scoring (out of 100) against the proposal assessment criteria set out in the national program guidelines.

Once the initial review was completed, Education Branch then assigned each proposal to two other regions (or to HQ officials within Education Branch) to conduct a second and third review (i.e. a peer review). The scores from all three reviews were then averaged for final score. Final funding decisions were then based on the final scores as well as other factors (such as whether the department had previously committed to funding a program that has existing students).

Throughout the process the NSC held regular meetings via conference call in order to discuss projects and differences in scoring. The quorum for an NSC meeting was 5 RDGs, and must include at a minimum those RDGs for which proposals are being challenged.

In total, for the 2014/2015 proposal review process, the Department received 220 PSPP proposals. Of those, 213 were subjected to three reviews, and 87 were approved for funding. Table 1 below provides a breakdown of regional and HQ involvement in the process in terms of the number of proposals received, the number of proposals not reviewed, the number of proposal reviewed, and the number of proposals approved. It highlights the unusually small number of proposals that were not subject to review (3%) based on program eligibility. It also highlights that collectively, HQ and Regions conducted 639 reviews. Interviews with regional staff indicated that the time required to complete the review of one project proposal varied based on factors such as the amount of material provided by the proponent for each project. With this variability in mind, regional staff indicated that the average time to simply review and score a single proposal was approximately 2.5 hours, not including the administrative time to manage the large volumes of information/documentation or the senior level briefings associated with the reviews. Based on 639 reviews, this suggests a total effort of over 213 person days dedicated to proposal review.

The audit noted that due to the call for proposals for 2014-15 PSPP funding not being initiated until February 2014, combined with the significant time investment associated with the NSC proposal review process, many successful proponents did not receive their funding until well after their projects began in September 2014. Program management has since addressed the timing issue for 2015-16 by issuing the call for proposals during the fall of 2014.

In its examination of the proposal review process, the audit team expected that a formal process would be established and documented, guidance materials would be developed to assist staff to make assessment decisions, and an analysis would have been undertaken to determine the amount of resources required to conduct the review.

The audit found that the process is clearly described in the NSC Terms of Reference, and guidance materials in terms of scoring assessment templates were also provided to regional staff to assist in the reviews. While some regional staff raised concerns around potential subjectivity in scoring, program management is continuing to refine the templates to minimize subjectivity.

The audit also found that the proposal review process was not supported by a resource requirements analysis. Staff from all regions visited raised concerns over the significant level of effort required to complete the reviews within a tight timeframe. In some cases, the review process forced staff to drop other work for a prolonged period in order to the complete the reviews. Concerns were also raised about the amount of effort required at the senior management level (i.e. RDG). While it was regional program staff that conducted the reviews, RDGs had to be briefed on them prior to the regular NSC meetings.

Based on the audit observations, the level of effort and resources required to undertake the proposal review process appears to not be aligned with the relative materiality and risk associated with the program.

Recommendation:

4. The Assistant Deputy Minister of ESDPP should clarify and confirm PSPP Program objectives in terms of their priority and alignment with broader Departmental priorities. Following this clarification and confirmation, the Program should

• Develop relevant performance indicators and align these with the Program DCI; and
• Identify and implement efficiencies in proposal review process to ensure that the resources and effort spent is commensurate with Program's significance, materiality and risk.

5.3.1 Improve the Program's performance measurement framework (2009 Internal Audit, Recommendation #4):

The audit identified that Program Management had implemented enhancements to the PMS for the PSSSP/UCEP Program which included the development of new performance indicators. Examination of these indicators demonstrated an appropriate linkage to the Program's expected outcomes. Further, as referenced in Section 5.1.3, it was determined that the information requirements within the PMS reflected alignment with the updated DCI.

While there have been improvements to the Program's performance measurement framework the substance of the 2009 recommendation was that the Performance Measurement produces data to permit adequate and appropriate analysis evaluation and reporting. As noted in Section 5.1.3, performance reporting is not in place as the EIS system has not been fully implemented for PSE. In the absence of this reporting, the Program has yet to establish performance targets nor have performance results been reported for PSE. As part of Management's response to the recommendation in 5.1.3, there is an expectation that performance reporting will support development of performance targets.

5.3.2 Improve clarity of Program requirements and delivery standards for recipients and for AANDC personnel involved in monitoring of these requirements (2009 Internal Audit, Recommendation #6):

The follow-up of this recommendation identified that Program requirements and delivery standards are now clearly set out in the national program guidelines. These guidelines were found to be updated annually and there was evidence of periodic improvements such as the addition of limitations regarding the administrative expenditures. The program guidelines are shared with appropriate Departmental staff and are posted to the departmental website. When requirements associated with the national program guidelines were changed, notifications are provided to recipients. In addition, some AANDC regional offices have supplemented the program guidelines with region-specific guidance which is provided to recipients and is also updated regularly.

Based on the follow-up procedures, Management has appropriately and sufficiently addressed this 2009 recommendation.

5.3.3 Establish a management control framework (2009 Internal Audit, Recommendation #7):

The 2009 audit recommendation indicated an expectation for a Program-specific Management Control Framework which included, as a minimum, the following:

1. A process for regularly updating the foundations of the Program, such as the Program T&C s, the Program Policy, the PSE Guidelines, and possibly an Operations Manual, to ensure that these documents remain relevant and sufficiently comprehensive;
2. An annual planning process that establishes objectives and targets for the Program each year, with a view to the continuous improvement of Program results (through the measuring and assessing of Program performance) and the enhancement of Program efficiencies;
3. The provision of guidance, direction to and oversight of regional offices through the provision of support tools, regular communications, operational guidelines, and well defined and communicated roles and responsibilities;
4. A resource planning process that considers the level of personnel needed to administer the Program, to meet Program objectives, and to ensure sufficient Program and agreement monitoring and oversight; and
5. A risk management process that identifies Program risks on an annual basis, provides a strategy for mitigating such risks on an ongoing basis, and ensures regular monitoring and sufficient risk based compliance auditing of recipients.

Follow-up procedures on these elements indicated that the key components of the Program's current management control framework consists of various guidance documents, such as the NPGs, some region-specific guidance, management meetings, and the provision of guidance to regional offices. In assessing these components, the audit identified that items (i) and (iii) as described above had been adequately addressed. However, there were ongoing gaps in terms of the following:

Annual Planning Process – As noted in Section 5.3.1, performance targets were yet to be established in connection with the updated Performance Management Strategy, as such there continues to be a gap in respect of this previous audit recommendation. The recommendation provided in Section 5.1.3 addresses this item.

Resource Planning Process – Follow-up procedures did not identify evidence that the Program had developed or implemented a planning process that considered the level of personnel needed to administer the Program, to meet Program objectives, and to ensure sufficient Program and agreement monitoring and oversight. As referenced in Sections 5.1.3 and 5.2, resource alignment concerns have impacted monitoring activities and efficiency of PSPP delivery respectively.

Risk Management Process – Follow-up procedures revealed the existence of a branch wide risk assessment covering all education programs. There was, however, no PSE program-specific risk management process that would, among other things, support development of strategies for risk mitigation, including risk-based performance monitoring and compliance activities.

The existence of ongoing gaps in the management control framework supporting PSE programs not only increases the risk that program objectives will not be achieved in accordance with requirements, but also the risk that prevailing management practices are ineffective, inconsistent and/or inefficient.

The audit team recognized that the Audit the Management Control Framework for Grants and Contributions (2013-14) was completed recently, and included a recommendation relevant to these findings. The objective of that audit was to assess the adequacy and effectiveness of departmental processes in supporting the design and approval of risk-based program control frameworks. It included an audit recommendation that the Department "…review and clarify departmental processes, governance structures, accountabilities, responsibilities and authorities for developing and approving program control frameworks…". Notwithstanding the over-arching nature of that previous audit recommendation, this audit included an assessment of the current state of the PSE Program's Management Control Framework and it is expected that the program can address the control gaps identified in the recommendation below in a timely manner, while also participating in the management actions to be implemented in response Audit the Management Control Framework for Grants and Contributions (2013-14).

Recommendation:

5. The Assistant Deputy Minister of ESDPP should strengthen key elements of control to support overall governance of the PSE Programs. In particular, it is recommended that:

• A Program-specific risk assessment should be completed by Education Branch on PSE programs upon which to implement risk-based controls to mitigate identified risks;
• A resourcing plan be developed to align the required level of resources with the operational needs for program delivery and monitoring activities.