Audit of the Management Control Framework for Grants and Contributions 2013-14 Focus on Program Control Frameworks and Recipient Reporting

Background

The Audit and Evaluation Sector of Aboriginal Affairs and Northern Development Canada ("AANDC" or "the Department") identified an Audit of the Management Control Framework for Grants and Contributions in the Department's 2013-14 to 2015-16 Risk-Based Audit Plan, approved by the Deputy Minister on February 27, 2013. A horizontal audit of the management control framework for grants and contributions management has been completed each year since 2010-11. The scope of these audits varies each year based on an assessment of risks during the planning phase of the audit and is designed to focus on a selection of key controls from the Department's Management Control Framework for Grants and Contributions. This year's audit focused on the design, approval and implementation of program control frameworks and on recipient reporting requirements.

AANDC makes funding available to First Nations and other recipients through Grants and Contributions (G&C) for the delivery of programs and services, including education, land management, social development and community infrastructure. Total departmental spending on G&C was $6.4 billion, $6.7 billion and $6.5 billion for the fiscal years 2011-12, 2012-13 and 2013-14, respectively.

AANDC's transfer payment programs are administered in accordance with the Treasury Board Policy on Transfer Payments and Directive on Transfer Payments, which took effect on October 1, 2008. The Policy on Transfer Payments outlines the expectations that risk-based approaches are adapted to the design of transfer payment programs, the preparation of terms and conditions, and funding agreements, and recipient monitoring and auditing. The objective of the Policy and Directive is to manage transfer payment programs with integrity, transparency and accountability, taking into account the risks, and to ensure that programs are effectively focused on citizens and beneficiaries, and are designed to achieve various Federal Government priorities and expected results.

In order to meet the expectations of the Policy and Directive, the Chief Financial Officer (CFO) Sector established the Transfer Payments Centre of Expertise, which has put in place the Management Control Framework (MCF) for G&C in order for the Department to effectively manage and monitor G&C programs and to ensure compliance to the Policy and Directive. AANDC's CFO is accountable for the overall management of transfer payment funds and, as such, is the custodian of the MCF for G&C.

The MCF for G&C establishes roles and responsibilities for the delivery of G&C, specifically to program management (the design and implementation of a program) and transfer payment operations (operations of a program with recipients). The MCF for G&C represents the departmental expectations of how G&C are to be managed across regions and at headquarters, and includes controls grouped into the following four areas: program design and approval; program monitoring and reporting; funding agreement development; and, transfer payment monitoring and reporting.

Audit Objective and Scope

The objectives of the audit were to assess:

1. the adequacy and effectiveness of departmental processes in supporting the design and approval of risk-based program control frameworks; and
2. the adequacy and effectiveness of controls governing and supporting the collection and use of recipient reporting.

The scope of the audit covered the period April 1, 2011 to December 31, 2013 and included assessments of:

• departmental processes which govern the development and approval of program control frameworks, giving particular consideration to whether these frameworks consider program and recipient risks and if they are designed to achieve maximum integration with other AANDC transfer payment management processes, tools and systems;
• departmental processes for managing the design, approval and rationalization of recipient reporting requirements, while ensuring that sufficient information is being gathered to support the Department's stewardship and accountability reporting obligations;
• processes for collecting, reviewing and analyzing recipient reports for a sample of three First Nation-focused programs (see Section 3.1 for an overview of the methodology used to select the sample);,
• processes for using information gathered from the analysis of recipient reports in making risk-informed decisions, including decisions related to: developing future agreements and defining the nature of the relationship with the recipient; providing support to First Nation recipients; monitoring implementation of First Nation programs and fulfillment of agreement obligations; and, increasing or decreasing future reporting obligations; and,
• regional delivery structures and human resource levels for the administration of grant and contribution programs in AANDC southern regions.

Statement of Conformance

This audit conforms with the Internal Auditing Standards for the Government of Canada, as supported by the results of the quality assurance and improvement program.

Observations

The Department has made progress in recent years in developing national policies for programs which aim to reduce variability in program delivery and funding approaches and streamline recipient reporting requirements. Regions are each proactively adjusting their organizational structures and internal role assignments to meet changing program and systems requirements; however each region is evolving independent of one another, making implementation of HQ-driven program control frameworks and systems challenging. An added complexity is that regions have made different levels of investment in program delivery staff and agreement management staff, contributing to variability in program implementation.

While the three programs examined had introduced some of the elements that would be expected in a strong program control framework, they included few risk-based approaches and generally lacked materials and training to support regions and recipients in implementation. In some instances, a lack of risk-based management practices led to over-control of low risk projects and recipients, while in other situations it manifested as insufficient control.

For the past two years the Department has been, and continues to be, focused on streamlining performance measurement strategies and recipient reporting requirements. Effective for the 2014-15 fiscal year, this will have resulted in a significant reduction in the number of reports being requested of recipients. While the Department has made progress in reducing recipient reporting requirements, the audit found that the data collected from recipients is not yet being used to its full potential by regions and programs. Programs and regions are working on rolling-out several new or improved program information systems aimed at centralizing data collection and supporting performance measurement. While implementation challenges associated with these systems has resulted in unexpected burden for recipients and regions, they are leading to clear improvements in the completeness and accuracy of data being collected.

Conclusion

The audit found that the Department does not take a horizontal approach to designing, approving or implementing program control frameworks. While the three programs examined had introduced some of the elements that would be expected of a program control framework, there was considerable opportunity to improve the consistency of approach across programs and the thoroughness of implementation in regions.

Although the audit found that the Department has made progress in reducing recipient reporting requirements for the programs included in the scope of the audit, the data collected from recipients is not being used to its full potential, nor are risk-based approaches being consistently used to target attention at areas of greatest need (e.g. limited risk-based reporting, risk-based compliance reviews and risk-based investments in community development / case management).

While regions are proactively adjusting their organizational structures and internal role assignments to meet changing program and systems requirements, each region is evolving independent of one another, making implementation of HQ-driven program control frameworks and systems challenging. An added complexity is that regions have made different levels of investment in program delivery staff and transfer payment management staff, resulting in varying degrees of program implementation across regions.

Recommendations

The audit team identified areas where management control practices and processes could be improved, resulting in the following four recommendations.

1. The Chief Financial Officer and the Senior Assistant Deputy Minister, Regional Operations, in collaboration with the Assistant Deputy Minister, Northern Affairs Organization and program Assistant Deputy Ministers, should review and clarify options within existing departmental processes, governance structures, accountabilities, responsibilities and authorities for developing and approving program control frameworks and establish a single window approach to communicating program control frameworks to regions and recipients.
2. The Chief Financial Officer and the Senior Assistant Deputy Minister, Regional Operations, in collaboration with the Assistant Deputy Minister, Northern Affairs Organization and program Assistant Deputy Ministers, should establish a function(s) to provide program design and program management expertise to HQ programs that are developing and implementing new and amended program control frameworks. This function could include a blend of existing expertise in program design and regional implementation with expertise in the development of risk-based program management regimes.
3. The Chief Financial Officer and the Senior Assistant Deputy Minister, Regional Operations, in collaboration with the Assistant Deputy Minister, Northern Affairs Organization and program Assistant Deputy Ministers, should improve alignment of program Data Collection Instruments with program Performance Measurement Strategies for their respective programs to further streamline data being collected from recipients. This should include delineating information required for performance measurement from information being collected for possible compliance activities to allow for the application of risk-based reporting regimes (e.g. for projects, programs and recipients).
4. The Chief Financial Officer and the Senior Assistant Deputy Minister, Regional Operations, in collaboration with the Assistant Deputy Minister, Northern Affairs Organization and program Assistant Deputy Ministers, should review key transfer payment management functions to promote greater consistency across regions, including regional organizational structures, classifications, capacity levels and role assignments.

Management Response

Management is in agreement with the findings, has accepted the recommendations included in the report, and has developed a management action plan to address them.

2.1 Audit Objectives

The objectives of the audit were to assess:

1. the adequacy and effectiveness of departmental processes in supporting the design and approval of risk-based program control frameworks; and,
2. the adequacy and effectiveness of controls governing and supporting the collection and use of recipient reporting.

2.2 Audit Scope

The scope of the audit covered the period April 1, 2011 to December 31, 2013 and included assessments of:

• departmental processes which govern the development and approval of program control frameworks, giving particular consideration to whether these frameworks consider program and recipient risks and are designed to achieve maximum integration with other AANDC transfer payment management processes, tools and systems;
• departmental processes for managing the design, approval and rationalization of recipient reporting requirements, while ensuring that sufficient information is being gathered to support the Department’s stewardship and accountability reporting obligations;
• processes for collecting, reviewing and analyzing recipient reports for a sample of three First Nation-focused programs (Capital Facilities and Maintenance Program; First Nations Child and Family Services Program; and Post-Secondary Education Programs). See Section 3.1 for an overview of the methodology used to select the sample;
• processes for using information gathered from the analysis of recipient reports in making risk-informed decisions, including decisions related to: developing future agreements and defining the nature of the relationship with the recipient; providing support to First Nation recipients; monitoring implementation of First Nation programs and fulfillment of agreement obligations; and, increasing or decreasing future reporting obligations; and,
• regional delivery structures and human resource levels for the administration of grant and contribution programs in AANDC southern regions.

3.1 Selection of Programs to Audit

The first element of the sampling methodology considers the programs to be selected for testing. Factors in the selection of the programs to be sampled included the following:

• Size (dollar value) of programs;
• Recentness of program design/redesign initiatives; and,
• Coverage across program sectors.

Based on the above analysis and with the objective being to assess program control frameworks and recipient reporting requirements horizontally across regions and programs, the following programs were selected as part of the conduct of on-site field testing at HQ and in the regions:

• Community Infrastructure Program (Capital Facilities and Maintenance Program);
• First Nations Child and Family Services Program (FNCFS program); and,
• Post-Secondary Education Programs (Post-Secondary Partnerships Program ("PSPP"), Post-Secondary Student Support Program ("PSSSP") and the University and College Entrance Preparation Program (UCEPP)).

3.2 Selection of Regions for Site Visits

The second element of the sampling methodology considered the regional offices to be visited. Factors in the selection of the regions to be visited included the following:

• Size (dollar value) of selected programs (selected above) funded by regional offices; and,
• Feedback obtained during the planning phase by individuals interviewed at HQ and regions.

Based on the above analysis and with the objective being to assess program control frameworks and recipient reporting requirements horizontally across regions and programs, the following regional offices were selected for on-site field testing:

• Ontario (January 10-14, 2014);
• British Columbia (January 27-31, 2014); and,
• Manitoba (February 10-13, 2014).

To gain an understanding of how all regions are adapting their organizational structures and staff roles, we expanded our interviews from the three regions visited to also include teleconferences with senior departmental officials in the Alberta, Saskatchewan and Quebec regions. The audit also included some telephone discussions with a selection of officials from First Nation communities, identified in consultation with regional offices.

5.1 Design of Program Control Frameworks

The audit included examination of the design and approval processes used by three programs which have recently undergone changes to their program control frameworks. The three programs and the changes undertaken in each include:

• Community Infrastructure Program (CI) updated its program control framework for major capital investments in 2009, moving to a national First Nations Infrastructure Investment Plan and introduced the Integrated Capital Management System (ICMS) to automate business processes and improve tracking of First Nations' infrastructure inventories and conditions. In 2013, an update was undertaken of the CI sub-program that funds assessments of the conditions for First Nation infrastructure assets (Assets Condition Reporting System (ACRS) assessments).
• First Nations Child and Family Services (FNCFS) Program is one of several social programs funded by AANDC. In 2012-13, the program control framework for all social programs was updated, and some national requirements were instituted for FNCFS, including the introduction of a FNCFS Annual Business Plan and adjustments to the Consolidated Annual Report for all child and family services agencies. FNCFS is also in the process of introducing a new national system to capture program data (FNCFS Information Management System).
• Audit of the Management Control Framework for Grants and Contributions 2013-14 Focus on Program Control Frameworks and Recipient Reporting, which funds post-secondary institutions to design and deliver university and college level courses tailored for First Nations and Inuit students, was updated for an April 2014 roll-out. Changes included a streamlined proposal intake and assessment process and a focus on programming in areas with high demand for labour. Also, a new information system and reporting regime was implemented in 2013-14 for the Post-Secondary Student Support Program (PSSSP) and the University and College Entrance Preparation Program (UCEPP). The Education Information System includes, among other things, a database for entering information collected on First Nations students who receive funding to pursue post-secondary education.

In each program control framework, we expected to find policy requirements, processes, implementation aids for regions and recipients, risk-based decision making frameworks, learning and development strategies and plans, recipient compliance monitoring processes, analysis and reporting to support regional program decisions, and a regime of quality assurance and continuous improvement activities being performed by the HQ program. A table showing the expected elements of a program control framework are depicted Exhibit 1.

As detailed in Table 2, we found that all three programs had significant gaps in their program control frameworks. While all three programs have established national-level policies that were subjected to senior-level review and approval, they were generally lacking in most other areas. Community Infrastructure had a reasonably complete control framework for its major capital projects and triennial assessments of on-reserve infrastructure (ACRS), but its operations and maintenance and minor capital projects were not well covered. The FNCFS program has introduced a risk-based compliance monitoring tool; however, we found that it had not been implemented in the regions we visited. FNCFS also funds evaluations of FNCFS agencies on a three-year cycle with a view to supporting continuous improvement. While these reviews were being completed by agencies, we did not see strong examples of the results being used to inform program decisions at the AANDC regional and program levels.

Based on our review of the design, change and approval processes applied to each program and our discussions with departmental officials, we determined that processes and responsibilities for undertaking program changes lack clarity. Firstly, involvement of the Chief Financial Officer (CFO) Sector, Regional Operations (RO) Sector, and other corporate functions varied from program to program. Secondly, it remained unclear as to who was responsible for supporting and challenging the majority of program control frameworks. While CFO Sector's Transfer Payments Centre of Expertise has core competencies in the areas covered by its financial- and agreement-focused directives, their expertise is admittedly not in program design and implementation. Likewise, while RO Sector's Planning and Business Integration Directorate has some capacity to work with programs undertaking major changes, they lack capacity and depth to support horizontal implementation and are not well positioned in the organization to play a challenge function on new and augmented program control frameworks.

All program and regional managers interviewed expressed the need for evolving a common approach across the Department with respect to the development and implementation of program control frameworks. This includes having common elements (e.g. policy, directives, processes, funding approaches, risk-based agreement management regimes, guides, tools, learning, program monitoring, recipient compliance, quality assurance and continuous improvement, etc.), common look and feel, common communication platforms for dissemination to regions and First Nations and a consistent annual cycle for rolling-out changes to regions and First Nations. The audit found that, in the absence of a clear departmental approach, programs concentrated predominantly on developing policies and rules providing limited support to regions and First Nations when they implement the programs. Some regions had evolved strong processes and guidance, but there was inconsistency in approach across regions visited for the programs included in the scope of our audit.

Through review of program design processes and interviews we noted that, while programs have traditionally invested most of their salary budgets in policy development expertise, some are moving to increase their program implementation expertise. However, all programs examined in our audit could benefit from shifting resources to hands on implementation expertise and making better use of regional expertise to design practical processes, guides, tools and training materials in support of program design and implementation.

5.2 Regional Delivery Structures and Staff Roles

Our audit found that regions have been adapting their delivery structures and internal staff role assignments to address changing program control frameworks, systems, and transfer payment management processes being rolled-out by HQ. To gain an understanding of how all regions are adapting their organizational structures and staff roles, we expanded our interviews from the three regions visited to also include teleconferences with senior officials in the Alberta, Saskatchewan and Quebec regions.

We found that each region is adapting its delivery model in different ways, with most regions moving responsibilities traditionally performed by Funding Services Officers (FSOs) to other agreement and program administration personnel (i.e. agreement development, management of recipient budgets and payment schedules, entering recipient reports into systems, assessing recipient reports to determine whether minimum program requirements have been met, and monitoring recipient compliance). In most regions, the FSOs continue to be the first line of communication with First Nations, are responsible for completing general assessments of recipients, work with communities in crisis, work with recipients in developing default remediation plans, and follow-up with recipients when reports are submitted late. In some regions, the FSOs retain some agreement management and program management responsibilities (e.g. CFS in Manitoba, minor capital projects in Ontario, capacity development programs in Quebec, assessment of select program reports in some regions, etc.).

The extent to which regions have established dedicated business units and created positions for agreement and program administration tasks appears to be linked in part to their relative size, with smaller regions having less executive and manager positions within which to manoeuver (i.e. maintaining appropriate spans of control for managers and Directors has an impact on organizational design). Appendix E includes an analysis of where the six types of regional staff are located in their respective regions.

The audit found that the differences in regional delivery structures and staff roles add complexity to the roll-out of program control frameworks, new systems, reporting changes and updates to transfer payment processes (e.g. compliance activities, recipient reinvestment plans for surplus funds on fixed agreements, etc.) We found that HQ program managers and other HQ-based functions are not always clear about whom to consult or brief in regions on changes and ongoing implementation issues, often providing briefings to regional program managers when it is the FSOs, heads of data units, or transfer payment management personnel who would benefit most from being consulted or briefed. We also found that the position classification categories (e.g. FI vs. AS vs. CR) and/or classification levels (e.g. AS-05 vs. AS-02) varied from region to region for certain roles, making it challenging for programs and the CFO Sector to gauge the level of guidance, training and oversight required by regional staff (see Table 4 later in this section).

To better understand the capacity of regions to implement program control frameworks, we analyzed each region's organizational design, resourcing levels^{Footnote 1} and role assignments, focusing on three program areas^{Footnote 2}: social programs; education programs; and, infrastructure programs. For purposes of our analysis, we grouped staff into six categories (Program Officers, Funding Services Officers, Other Transfer Payment Management Personnel, Data Entry Clerks, Community Development Officers, and Compliance Officers) according to the functions they perform. These functions do not necessarily equate to position titles or organizational units, which bare similar titles. For example, positions in Education directorates which are focused on data entry or compliance are included in the Data Entry Clerks and Compliance Officers categories, while program analysts, program officers, managers, directors, and administrative staff in education directorates are included in the Education Category. This distinction was necessary because organizational design and allocation of responsibilities differs from region to region.

Table 3 compares the resourcing levels for each of the program and agreement management functions and highlights that certain regions have invested more resources in some functions than others. Considering that regional program management staff with responsibility for supporting education and social programs are typically assigned multiple programs, we were unable to delineate our data beyond the program cluster (i.e. figures presented in the table for "Education" and "Social" include resources devoted to all education and social programs).

Table 4 below shows the classification categories and levels for transfer payment management and administration personnel who perform agreement management tasks such as agreement development, budget management, and quality reviews. An interesting note is that the Alberta region leverages financial professionals to support financial related aspects of agreement development and management whereas other regions employ administration professionals and clerical personnel for these functions. Additionally, some regions tend to rely more heavily on positions with higher classification levels (e.g. Quebec and Saskatchewan).

Considering that recipient demographics vary across regional offices, we assessed regional capacity according to the following three other factors: the amount of G&C program dollars administered by the region^{Footnote 3}; the number of First Nations^{Footnote 4} and Tribal Councils with whom the region administers funding agreements; and, the percentage of recipients (First Nations and Tribal Councils) in the region that are in default of their agreements and are undergoing some form of remedial action. This analysis highlighted additional discrepancies in regional capacity levels and is included in Appendix C.

5.3 Implementation of Program Control Frameworks in Regions

In section 5.1, we presented our findings related to the design of new and amended program control frameworks and highlighted that elements of the frameworks intended to support regional and community level implementation of programs were lacking. In section 5.2, we presented our analysis and findings on regional delivery structures and highlighted how they vary considerably across regions. In this section, we examine how these two findings intersect and impact the effectiveness of the implementation of program control frameworks.

In general, we found that regions have adapted different approaches to implementing the three programs examined in the scope of the audit. Considering that FNCFS programming follows provincial models and standards, comparisons across regions are not meaningful. Accordingly, we focused our comparative analysis on the Community Infrastructure and Post-Secondary Education programs, which are not striving to mirror provincial models and do permit meaningful comparison across regions.

5.4 Collection and Use of Recipient Reporting

In addition to assessing the design and approval of new and modified reporting requirements, covered in section 5.1 as part of our assessment of the design and approval of program control frameworks, the audit assessed the Department’s controls for the collection and use of recipient reporting. To this end, we performed sample testing in three regions to examine whether regional personnel are using recipient reports for recipient-level decision making. We also examined the issue from the perspective of the HQ program and regional program managers to determine whether they are supporting regional personnel with the necessary processes, training and systems to enable the effective and efficient use of reporting.

We found that the Department has been actively engaged over the past three fiscal years in evaluating the need for the data being collected from recipients and introducing new program information systems (Integrated Capital Management System (ICMS), Education Information System (EIS) and FNCFS Information Management System (IMS)) to streamline collection and processing of recipient data. Based on information obtained from departmental officials, we understand that the number of Data Collection Instruments collected by AANDC has been reduced considerably for the three program areas in question as a result of a commitment by the Government of Canada to reduce recipient reporting burden.

Table 6 shows that the number of distinct reports requested of recipients, of all reporting intervals, has been reduced by approximately two-thirds between 2011-12 and 2014-15. Note, however, that some of the reports required in prior years were merged over the period in question, so the reduction in the number of reports may be higher than the reduction in data points being collected. We were not able to obtain an analysis of the number of data points being collected year-over-year.

At the time of our audit, two of these systems were in the process of implementing and debugging their data collection modules (EIS and FNCFS (IMS)) and had not yet implemented advanced data analysis and reporting functionalities. The data collection modules for the third system, ICMS, were fully implemented in all regions visited and some system-generated reports were available to assist in managing recipient agreements. Based on our review of a sample of reports in each region and discussions with regional and HQ program officials, we found that all systems are leading to marked improvements in the completeness and accuracy of data collected from recipients and entered into systems. This focus on data completeness and accuracy has translated into an increased demand on many First Nations who may not have been submitting complete or accurate reports in prior years.

Based on our review of documentation and discussions with region and HQ program staff, we observed that program information systems included more focus on gathering performance information for HQ programs and less focus on designing reporting for regional program delivery staff (e.g. there are few standard reports for program managers, program advisors and recipients, which are intended to support ongoing management of the program). We identified opportunities in all programs to leverage the information systems and recipient data to improve risk-based decisions regarding the various agreement management approaches. For example, ICMS includes tracking of what infrastructure is located in each community and the results of triennial inspections of infrastructure; together this information could be used to assess whether a given community’s infrastructure is depreciating at an expected rate and could be used to inform risk-based monitoring, management and investment decisions. As another example, the post-secondary module of EIS includes collection of transactional information that could be used to run automated scripts designed to identify potential anomalies (e.g. students funded from two sources, First Nations communities with inconsistent per student funding levels, First Nations who are not reporting sufficient transactional data to account for all of their funding, etc.).

Our regional fieldwork highlighted that AANDC FSOs and Data Entry Clerks devote considerable amounts of time to following up with recipients on late reports. We extracted information from the AANDC Grants and Contributions Information Management System (GCIMS) on all reports requested of and received by recipients in 2012-13. We found that only 17% of 2012-13 AANDC reports were received on time, 68% were received late, 9% had not been received as of February 1, 2014 and 7% had been cancelled and deemed unobtainable. As detailed in Exhibit 2, these results varied only slightly for the three main program clusters analyzed.

We also evaluated regional reporting to determine whether regions had different levels of success in collecting reports on time. Exhibit 3 shows that while some regions have slightly better experiences than others, late reporting and cancellation of reports is a problem across the country.

We performed additional trend and correlation analysis to determine whether reporting lateness was higher or lower when considering a recipient’s general assessment score, Community Wellbeing Index score, and population size, but found no strong correlation with any of these factors.

The high frequency of reports that are cancelled and/or not received, coupled with our testing observations that regional personnel are not consistently using data from reports to make risk-based decisions, highlights the importance of re-evaluating whether the amount and type of information identified for collection is actually needed. At the time of writing our report, the Department was in the process of rolling out streamlined Data Collection Instruments (DCIs) to reduce the number of reports required of First Nations. The reductions are reflected in Table 6, found above in this section.

5.5 Horizontal Findings and Recommendations

On the basis of our findings and analysis presented in sub-sections 5.1 to 5.4, we concluded that there are five systemic challenges or barriers that need to be addressed in order to improve the effectiveness of program control frameworks, which are:

• requirements and processes governing the design and approval of Program Control Frameworks are unclear;
• program control frameworks include few risk-based reporting and management regimes;
• recipient reports do not delineate between information required for performance measurement and information required to administer recipient agreements;
• gaps in governance processes for new information systems lead to implementation challenges that could have been avoided; and,
• inconsistencies in regional delivery models and capacity levels lead to inconsistent implementation of program control frameworks.

5.5.3 Delineation of Recipient Reporting Requirements

The audit included examination of the Performance Measurement Strategies (PMS) and Data Collection Instruments (DCIs) of the three programs included in the audit scope. We found that most of the data points in the DCIs were not directly supporting performance indicators in the PMSs. For two of the three programs included in the audit, infrastructure and education, we noted that most of the data collection strategies for measuring performance indicators in the program PMSs called for obtaining data from alternate sources, many of which had not yet been defined. Since January 2011, social programs have been updating their performance indicators and DCIs to ensure that data is collected from recipients where practical and used to report on program performance. During the period of the audit, a departmental initiative was underway to streamline PMSs and reduce insignificant indicators and data collection requirements.

Description of Exhibit 4

The bar graph shown in exhibit 4 demonstrates the amount of essential and non-essential for the Community Infrastructure, Education and Social Programs for 2012-2013. The bar graph shows that Social Program has had the most essential spending of three programs listed. On the contrary, the Education Program has the most non-essential spending of the three programs.

Community Infrastructure (all amounts in Canadian dollars)

Essential Spending: 502,000
Non-Essential Spending: 538,000
Total Spending: 1,040,000

Education Program

Essential Spending: 1,030,000
Non-Essential Spending: 669,000
Total Spending: 1,099,000

Social Program

Essential Spending: 1,350,000
Non-Essential Spending: 344,000
Total Spending: 1,694,000

---

 

We also found that none of the reporting regimes for the three programs examined employed a risk-based approach. All data was requested of all recipients, regardless of their history of meeting obligations under the agreement or their demonstrated capacity. We found that when programs neglect to delineate performance measurement reporting from program management and compliance reporting, it impedes the regions’ ability to apply risk-based reporting regimes. Moreover, the heavy burden of entering, reviewing and approving recipient reports leaves regional staff with little time to perform program management and recipient monitoring tasks. It also negatively impacts the role of FSOs by preoccupying their time with chasing down late reports to avoid funding halts, notwithstanding that most core programs have been deemed essential (see Exhibit 4) which restricts funding from being halted.

Recommendation:

3. The Chief Financial Officer and the Senior Assistant Deputy Minister, Regional Operations, in collaboration with the Assistant Deputy Minister, Northern Affairs Organization and program Assistant Deputy Ministers, should improve alignment of program Data Collection Instruments with program Performance Measurement Strategies for their respective programs to further streamline data being collected from recipients. This should include delineating information required for performance measurement from information being collected for possible compliance activities to allow for the application of risk-based reporting regimes (e.g. for projects, programs and recipients).

I. Relative Capacity of Regions Based on G&C Program Dollars Administered

When analyzing regional capacity relative to the amount of G&C program dollars Regions administer, we found that regions place emphasis on different types of personnel capacity. For example, Exhibit 5 demonstrates that British Columbia and Ontario regions invest more resources in supporting infrastructure programming compared to other regions. Similarly, Exhibit 6 below shows that Atlantic, Quebec and Alberta regions have invested relatively higher resourcing levels in Education programming, while Exhibit 7 below shows that Atlantic, Quebec, Alberta and British Columbia regions invest in higher levels of capacity for social programs. Some of the variances in investments in social program staff attribute to provincial delivery models because some provinces and provincial organizations assume all or part of the responsibilities for some social programming (i.e., Ontario and British Columbia).

Exhibit 8 below shows that the Quebec and Ontario regions have relatively lower levels of Funding Services Officers when considering the amount of G&C funding administered. While Exhibit 9 below shows that the Manitoba and Alberta regions have relatively lower levels of other transfer payment management personnel (Data Operations, Transfer Payment Management, Compliance Officers, and Community Development Officers) when considering the amount of G&C funding administered. An important note is that we did not examine whether the lower capacity levels in the regions were as a result of the region opting to invest in other programs not included in our analysis (e.g. lands management, estates and trusts management and economic development), therefore we make no inferences about whether, on the whole, the region’s capacity is higher or lower than the capacity of other regions.

II. Relative Capacity of Regions Based on Proportion of Recipients in Default of Funding Agreements

One of the observations made consistently by regional staff is that more attention of FSOs and program officers is required when working with communities that are in default of their funding agreements with AANDC (as determined by the AANDC Default Prevention and Management Policy).

To better understand capacity from this perspective, we compared the percentage of First Nations and Tribal Councils in default in each region to the average number of staff^{Footnote 5} available to support every community (both in default and not in default). We expected to find that regions with greater frequency of First Nations in default (Ontario, Manitoba, Quebec and Saskatchewan) would have more resources dedicated to supporting these First Nations. However, we found little positive correlation between the proportion of recipients in default in a region and the region’s personnel capacity to support these First Nations (see Exhibit 10).

III Relative Capacity of Regions Considering Number of First Nations and Tribal Councils Served and Regional Staffing Levels

Regions with large numbers of smaller First Nations (i.e. small on-reserve community populations) sometimes observe that they have a disproportionately high number of agreements and reports to administer and review. We believe that this argument has merit for programs that are funded directly to all or most communities (e.g. education and infrastructure operation and maintenance), but is less applicable to programs where only a portion of communities are funded, such as FNCFS programming and major capital infrastructure projects.

Exhibit 11 shows that some regions have invested considerably more salary dollars in education program staff per First Nation (Alberta, Atlantic and Quebec) than others. An interesting note is that the regions with the lowest number of First Nations served and the lowest amount of G&C dollars administered are the regions with higher relative capacity in their education program units, suggesting that regions have determined that they have a base capacity required to administer education programs.