Archived - Audit of the Quality Management Program and the Application of the Intervention Policy

Date: February, 2009

PDF Version (166 Kb, 40 pages)

Table of Contents

• Initialisms and Abbreviations
• Executive Summary
• 1.0 Introduction
• 2.0 Objectives
• 3.0 Scope
• 4.0 Approach and Methodology
• 5.0 Observations and Recommendations - Application of the Intervention Policy
• 6.0 Observations and Recommendations - Quality Management Program
• 7.0 Action Plan
• 8.0 Appendix A - Audit Criteria

Initialisms and Abbreviations

Executive Summary

Introduction

Grant and Contribution (G&C) programs are the primary instrument used by Indian and Northern Affairs Canada (the "Department") to deliver its mandate. G&C expenditures represent approximately 85% of the Department's budget.

The Risk-Based Audit Plan developed by the Audit and Assurance Services Branch of the Audit and Evaluation Sector (AES) identifies that a horizontal audit of controls associated with grants and contributions is to be conducted in each of the fiscal years from 2008/09 through 2010/11. The rationale for this multi-year audit focus is a result of the significance, complexity and sensitivity of G&C programs. The objective of a horizontal audit is to provide an assessment of management's controls across the Department, rather than to evaluate controls associated with a specific program, region or sector. The focus of this horizontal audit, the Audit of the Quality Management Program and the Application of the Intervention Policy (the "Horizontal Audit"), was directed toward two key Department-wide risks as they relate to Departmental G&C programs: (1) the Application of the Intervention Policy, and (2) the Quality Management Program.

Application of the Intervention Policy

The Intervention Policy ("IP") sets out a framework for intervention by the Minister in the event that a Recipient defaults under the terms and conditions (Ts&Cs) of a Funding Arrangement. According to this Policy, intervention is intended to "…permit the delivery of programs and services and maintain accountability while problem situations are being addressed and to put the onus on the Recipient to correct problem situations." The IP is designed to support timely intervention and consistency in regional operations as well as to facilitate ongoing monitoring of intervention and to improve the effectiveness of intervention. The Policy also states that "The aim of the policy is to encourage the Recipient in default to enhance their capacity to provide programs and services and to provide for an exit strategy where a lesser, or no form of Intervention is required." To achieve and sustain this important outcome in a consistent manner across all regions, management's controls around the application of the IP must be well-designed and effectively operationalized. This Horizontal Audit addresses these important controls.

Quality Management Program

A quality management program is a method used by management to ensure that necessary activities across all phases of product/service delivery are in place and effective. In the context of this Horizontal Audit, the Quality Management Program (QMP) describes the methods used by management at both Departmental Headquarters (HQ) and in the Regions to ensure the Department's ongoing compliance with G&C funding requirements. These requirements can be found within program terms and conditions, within specific funding arrangements or elsewhere (e.g. applicable Departmental and Treasury Board policies). For the purposes of this Horizontal Audit, the Quality Management Program has been split into two major categories of activities: Quality Control (QC) and Quality Assurance (QA). Quality Control activities are those intended to ensure compliance is maintained through the conduct of day-to-day operations. Quality Assurance includes those activities used by management to independently validate compliance with requirements (QA activities are performed by a party that does not have operational responsibility). Both activities are essential to an effective Quality Management Program. This Horizontal Audit addresses management's controls over both sets of activities.

Objectives, Criteria and Scope

The overall objective of the Horizontal Audit was to provide assurance to senior management regarding the application of the Intervention Policy and the Quality Management Program (including Quality Control and Quality Assurance activities). The Terms of Reference for this project outlined a number of related Grant and Contribution Audit Criteria[Note 1] as follows:

• Appropriate quality assurance policies and procedures and roles and responsibilities are developed, communicated and efficiently and effectively implemented to ensure that programs are applied consistently within approved terms and conditions and other relevant policy requirements;
• Risks to the achievement of program objectives are effectively mitigated through the application of quality assurance activities;
• The Intervention Policy is implemented and followed as appropriate; and
• Appropriate reduction targets are defined and the management control framework is adequate to ensure their achievement, including effective supplier direction and management.

Building on the above audit criteria, audit procedures conducted during the planning phase of this Horizontal Audit supported the clarification and fine-tuning of the audit objectives as follows.

1. Application of the Intervention Policy – Audit objective is to provide assurance to senior management that:

• Controls to ensure that regions/sectors are in compliance with the requirements outlined in the Intervention Policy are adequate and effective.

2. Quality Management Program – Audit objectives are to provide assurance to senior management that:

• Key management controls are adequate and effective to ensure compliance with program and agreement terms and conditions and applicable Indian and Northern Affairs Canada (INAC)/Treasury Board Secretariat (TBS) policies and procedures are adequate and effective (Quality Control); and
• There is an effective and independent program of Quality Assurance to objectively validate compliance with the terms and conditions of programs and of specific contribution agreements, and with applicable INAC/TBS policies and procedures (Quality Assurance).

Detailed audit criteria applicable to each of these audit objectives were also developed and are presented in Annex A.

The scope of the Horizontal Audit included all major grant and contribution programs, with emphasis on Education, Social and Capital programs, across regional offices. The focus of the audit was on current requirements and recent practices: as such, active or recently active G&C files were selected for audit testing. During the planning phase of this Horizontal Audit, interviews were conducted with management representing a sample of regions and sectors. During the conduct phase, on-site visits were completed at the 7 southern regions (based on materiality of program delivery in these regions) and follow-up interviews were held with management at HQ.

The approach to the internal audit followed the requirements of the TBS Policy on Internal Audit and the Institute of Internal Auditors' Standards for the Professional Practice of Internal Auditing.

Conclusions & Recommendation

Conclusions

The Horizontal Audit raised a number of conclusions regarding the application of the IP and the status of the Quality Management Program (QMP) over Grants and Contributions. The Horizontal Audit identified management control framework, capacity and capability issues that have contributed to significant variations (between policy and practice and from region to region) in how the IP is applied and with respect to the nature/reliability of Quality Management activities across the Department.

The audit identified a relatively common perception that the IP is not effective relative to objectives this policy is intended to promote. This perception appears linked to variances regarding the extent to which the IP is diligently applied.

The audit identified an overall and systemic weakness in the clarity and consistency of quality control and quality assurance roles and responsibilities within and across sectors, programs and regions. The audit concluded that there is an overall lack of formalization and enforcement of expectations relative to quality control and quality assurance. Specifically, it was noted that the Department's QMP was neither defined nor communicated to relevant stakeholders and that the approach to monitoring compliance was not coordinated between regions and HQ.

These observations support the conclusion that management's controls are not adequate to ensure:

• Ongoing compliance with program Ts&Cs and INAC/TBS policies;
• Effective monitoring, on a national scale, of the adequacy or effectiveness of management's controls as they relate to application of the IP, Quality Control or Quality Assurance; and
• The application of judgment or decision-making authority is appropriate, deliberate and consistent across the Department.

Audit testing of files supported these conclusions regarding weaknesses in management's controls surrounding ongoing compliance. For example, regarding the correct application of the IP, in 47% of the files tested, Remedial Action Plans (RMPs) were not obtained in a timely manner. Further, 72% of the files tested did not include adequate documentation to support the monitoring activities (a key element of quality control) required within program Ts&Cs.

While audit procedures revealed evidence of quality control practices, the overall conclusion is that there are significant gaps and weaknesses in management's controls with respect to each of the audit objectives. Until these issues are addressed, AES cannot provide senior management with assurance that controls intended to promote ongoing compliance with Ts&Cs and with INAC/TBS Policies (including the IP) are reliable.

Recommendations

In order to address the conclusions described above, recommendations have been developed that are intended to direct management to what we consider are the root causes of the control weaknesses identified within the audit.

With regards to the IP, management should undertake an assessment of the IP to determine and confirm its effectiveness. This assessment should address the role of the IP relative to the Departmental Mandate and include consideration of roles, responsibilities and expectations. The Department should promote and facilitate effective and efficient monitoring of the IP by establishing an integrated plan, using a risk based approach, to evaluate compliance with the IP at a National level.

With regards to the QMP, management should develop an integrated and standardized QMP that can be reasonably implemented and maintained across the Department. This Department-wide program should clearly identify roles and responsibilities of key stakeholders both in HQ and in the regional offices. Further, in connection with developing an integrated QMP, management should develop standard expectations, guidance and tools for the implementation of quality control activities at the regional level. These should include guidance regarding the monitoring, review and documentation of recipient program reports. Finally, the Department should establish a common and integrated policy that defines expectations for using a risk-based approach to G&C compliance reviews across the Department. This policy should address the need for a strategy and plan to ensure the resources and capabilities exist to implement these compliance reviews.

Details of these conclusions and recommendations are provided in Sections 5.0 and 6.0 of this report.

1.0 Introduction

Grant and contribution programs represent $5.5 billion in Departmental expenditures (over 85% of total expenditures) every year. The funds are transferred to approximately 1,200 entities, including about 640 First Nations and Aboriginal organizations. These transfers are crucial in the achievement of the Department's mandate. As such, the 2007-10 Risk-Based Audit Plan established a principle that an audit would be conducted annually to provide a horizontal (Department-wide) assessment of key grant and contribution controls.

As a result of recent internal audit activity and regional management practices reviews, the Audit and Evaluation Sector identified program QA and the application of the IP as key risk areas for audit in 2008-09.

Application of the Intervention Policy

Intervention, as defined in the IP is the "…exercise by the Minister of any of the remedies available to the Minister under a Funding Arrangement in the event of default by a Recipient…". The conditions of default may be related to issues concerning the recipient's financial, governance or other situation that could impact the recipient's capacity to deliver the programs and services provided via the funding arrangement. The IP is designed to support timely intervention and consistency in the response to conditions of default across regional operations. It also aims to encourage the Recipient in default to enhance their capacity to provide programs and services and to provide an exit strategy.

There are three levels of Intervention:

1. Recipient Managed is a low level of intervention which is applied when a recipient is determined to have both the willingness and capacity to resolve the difficulties that gave rise to the default.
2. Co-Management is a moderate level of intervention which is applied when a recipient is determined to have the willingness, however lacks the capacity to resolve the difficulties that gave rise to the default.
3. Third Party Management is the highest level of intervention which is applied when a recipient is determined to lack both the willingness and the capacity to resolve the difficulties that gave rise to the default.

Examples of concerns related to the Intervention Policy included, amongst others:

• About 50% of First Nations under intervention with no trend towards a reduction in the number of recipients under intervention;
• A large number of First Nations remaining in third party management for extended periods of time;
• Apparent inconsistencies between funding agreements and intervention status; and
• Apparent reluctance to use third party intervention even when significant problems exist.

Quality Management Program

A quality management program is a method used by management to ensure that necessary activities across all phases of product/service delivery are in place and effective. In the context of this Horizontal Audit, the Quality Management Program refers to the processes and practices used by management to ensure that the Department delivers and monitors programs and services in a manner that respects the program terms and conditions and funding arrangements and applicable policies or other requirements. For the purposes of this project, the QMP has been defined as comprising two major categories of activities: Quality Assurance (QA) and Quality Control (QC). QC activities are those that ensure compliance is reflected in all day-to-day activities (e.g. the documentation of files or the completion of forms that address compliance requirements associated with the funding arrangement). QA includes those procedures, practices and other controls management uses to independently validate the achievement of overall compliance (e.g. when a compliance unit or a HQ function conducts a review of file or other evidence to ensure that compliance requirement have been addressed). Both activities are essential to an effective Quality Management Program.

Examples of concerns related to the Quality Management Program (QMP) included, amongst others:

• Non-performance of required reviews;
• Absence of comprehensive QMP (QA and QC) strategies or plans;
• Overly detailed and potentially ineffective reviews; and
• Absence of risk-based, coordinated approaches to QMP (QA and QC).

2.0 Objectives

The overall objective of the Horizontal Audit was to provide assurance to senior management regarding the application of the Intervention Policy and the Quality Management Program (including Quality Control and Quality Assurance activities). The Terms of Reference for this project outlined a number of related Grant and Contribution Audit Criteria as follows:

• Appropriate quality assurance policies and procedures and roles and responsibilities are developed, communicated and efficiently and effectively implemented to ensure that programs are applied consistently within approved terms and conditions and other relevant policy requirements;
• Risks to the achievement of program objectives are effectively mitigated through the application of quality assurance activities;
• The Intervention Policy is implemented and followed as appropriate; and
• Appropriate reduction targets are defined and the management control framework is adequate to ensure their achievement, including effective supplier direction and management.

Building on the above audit criteria, Planning Phase audit procedures supported the clarification and fine-tuning of the objectives for this Horizontal Audit as follows:

1. Application of the Intervention Policy – Audit objective is to provide assurance to senior management that:

• Controls to ensure that regions/sectors are in compliance with the requirements outlined in the Intervention Policy are adequate and effective.

2. Quality Management Program – Audit objectives are to provide assurance to senior management that:

• Key management controls are adequate and effective to ensure compliance with agreement terms and conditions and applicable Indian and Northern Affairs Canada (INAC)/Treasury Board Secretariat (TBS) policies and procedures are adequate and effective (Quality Control); and
• There is an effective and independent program of Quality Assurance to objectively validate compliance with the terms and conditions of contribution agreements, and with applicable INAC/TBS policies and procedures (Quality Assurance).

3.0 Scope

The scope of the audit included all major grant and contribution programs, with an emphasis placed on Education, Social and Capital programs, across regional offices. The focus of the audit was on current requirements and recent practices, as such, active or recently active G&C files were subject to testing. The following is a description of the scope of audit activities for each of the areas of focus for this Horizontal Audit:

Application of the Intervention Policy

In terms of the IP, the audit focused on current requirements and recent practices at the regional level and included a case file review of a sample of First Nations including those currently operating under the IP and those that would appear to warrant consideration under the IP. To the extent that there were lessons to be learned or best practices to be identified, file reviews included a retrospective examination of the relationship between First Nations and the regions.

The audit also included an examination of the possible factors leading to First Nations remaining in third party management for extended periods of time and the potential opportunities for INAC to assume more proactive policies and practices to address such situations.

Quality Management Program

For the purposes of this Horizontal Audit, the QMP has been defined as comprising two major categories of activities: QC and QA. The audit examined the relevance and application of policies, procedures, tools, practices, and reports for major national programs and for all regional offices relating to both QC and QA. In-scope activities subject to audit testing addressed a number of criteria including:

• Practices and procedures to report on the status of recipient's compliance with requirements
• Tools and mechanisms to identify compliance issues and related escalation procedures
• Extent and nature of quality assurance reviews at a regional and national level
• Assessment of the independence of QA personnel

The audit of the QMP specifically excludes processes and controls related to non grant and contribution transactions (e.g. departmental finance, human resources, and contracting).

4.0 Approach and Methodology

The approach to the audit followed the requirements of the Institute of Internal Auditors' Standards for the Professional Practice of Internal Auditing and the TBS Policy on Internal Audit. This means that sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the conclusions reached and contained in this report. The conclusions are based on a comparison of situations, as they existed at the time of the audit and against the audit criteria. It should be noted that the conclusions are only applicable for the areas examined.

The planning phase of the audit involved various procedures including: documentation review, interviews with management, regional visits to Quebec and Saskatchewan and teleconferences with representatives from all other regions and of the following sectors: Lands and Trust Services, Claims and Indian Government, Socio-Economic Policy and Regional Operations (SEPRO) (now Education and Social Development Programs and Partnerships) and the Office of the Federal Interlocutor for Métis and Non-Status Indians (OFI).

The results of these planning procedures were synthesized and analyzed in order to develop draft Management Control Frameworks (MCFs) for both QA and the application of the IP. These MCFs articulated various control objectives as identified by regional and HQ management. Audit criteria were determined based on information gathered during the planning and risk assessment phase during the period of March through June 2008. The audit criteria served as the basis for developing the audit approach and detailed audit program for the conduct phase. The audit criteria are provided in Annex A.

During the conduct phase of the audit, the activities in 7 ("south of 60¡") regional offices were examined in detail. The 7 Southern regions were selected based on risk and materiality. The Northern regions were excluded from detailed testing based on the materiality of program delivery.

The principal audit techniques used included:

• Documentation Review – thedocumentation that was subject to examination included, but was not limited to,policies, directives, frameworks, and procedures relevant to Quality Assurance over G&C program delivery and the application of the Department's Intervention Policy.
• Recipient File Review – file reviews were conducted using an audit program developed to assess compliance with related audit criteria.
  • With respect to the application of the IP, file documentation was examined for a sample of 29 recipient funding arrangements in the 7 regions visited.
  • With respect to QA over G&C program delivery, file documentation was examined for a sample of 35 recipient funding arrangements in the 7 regions visited.
• Interviews – interviews were conducted at HQ with representatives from the Transfer Payment Directorate (TPD) and the Financial Policy Directorate. Furthermore, interviews were conducted at each of the 7 regions visited with management and staff responsible for the delivery and financial oversight of G&C programs. Interview guides were developed for interviews conducted, taking into consideration the objectives of the audit and the audit criteria developed.

Audit fieldwork was conducted in the regions and at HQ between August and November 2008.

5.0 Observations and Recommendations - Application of the Intervention Policy

Key conclusions

The effectiveness of the IP as a means to achieve intervention objectives is not universally acknowledged. For example, there is a perception that Third Party Management may have the unintended impact of promoting recipient dependency on the Department. With the effectiveness of the IP under challenge, there is an increased likelihood that regions will resist the diligent application of the policy in favour of the ad-hoc application of judgement and experience to individual recipient situations.

With the current IP, there is also no clear understanding of the distribution of roles and responsibilities across the wide range of activities that comprise the Department's intervention regime. This lack of clarity is a key contributor to many of the concerns that led to the delivery of this Horizontal Audit (e.g. 50% of FNs under intervention with no trend towards a reduction and a large number of FNs remaining under third party management for extended periods). Further, the resulting lack of understood and acknowledged expectations across key stakeholder groups including the CFO Sector, Programs and Regional Operations (particularly FSOs) has led to inconsistent practices and/or understandings relative to:

• How and when judgment may be applied when interpreting the terms and conditions of the IP; and
• IP design, development, implementation, and monitoring

There are also gaps and inconsistencies in the Quality Management Program (see Section 6.0) as they apply to the application of the IP. These gaps and inconsistencies reflect the absence of an integrated approach to monitor compliance and a limited capacity for such monitoring at both regional and national levels. Regional compliance activities were focused on recipient compliance rather than compliance with the IP while monitoring performed by HQ is limited, ad-hoc, and not risk based.

These observations support the conclusion that management's controls are not adequate to ensure:

• Ongoing compliance with the provisions of the IP;
• Effective monitoring, on a national scale, of the adequacy or effectiveness of management's controls as they relate to application of the IP; and
• The application of judgment or exercise of authority is appropriate, deliberate and consistent as it related to interpreting and applying the IP to individual recipients.

Section 5.1 and 5.2 of this report provides details on the observations and conclusions noted during the course of the audit with respect to the application of the IP and recommendations to assist the Department in addressing these observations/conclusions.

Background

The Intervention Policy sets out a framework for intervention by the Minister in the event that a Recipient defaults under the terms and conditions of a Funding Arrangement. According to the IP, intervention is intended to "…permit the delivery of programs and services and maintain accountability while problem situations are being addressed and to put the onus on the Recipient to correct problem situations." The IP is designed to support timely intervention and consistency in regional operations as well as to facilitate ongoing monitoring of intervention and to improve the effectiveness of intervention. The IP also states that "The aim of the policy is to encourage the Recipient in default to enhance their capacity to provide programs and services and to provide for an exit strategy where a lesser, or no form of Intervention is required." To achieve and sustain this important outcome in a consistent manner across all regions, management's controls around the application of the IP must be well-designed and effectively operationalized.

5.1 Application of the Intervention Policy

The Intervention Policy is not being diligently and consistently applied on a national basis.

While the IP is designed to allow for the application of judgment when considering the circumstances of an individual recipient, the IP is also intended to promote consistency in the application of intervention across the Department. These potentially conflicting objectives require management to engage tools and practices (e.g. formal criteria, guidance and training) that will promote an effective balance between judgment and consistency.

In the absence of a foundation to guide the application of judgment on a Department-wide basis, there is a considerable risk that the IP will not be applied consistently to otherwise very similar recipient situations. For example, in cases where the proportion of recipient funding provided via the Department is very low, there appeared to be less likelihood that the IP would be applied despite conditions of default.

The audit found differences in the interpretation of the IP which have led to variations in the application of the IP to otherwise comparable situations. Even with respect to clearly quantitative conditions of default (e.g. recipient's operating deficit exceeds the 8% threshold), the audit identified variations in the application of intervention. The underlying issue was linked to how regions interpret conditions of default as described in the IP. In some regions, certain conditions of default were viewed as triggers, which resulted in intervention. In other regions, the same conditions were viewed as indicators, which meant that intervention may (or may not) be applied.

The audit identified an absence of clear roles, responsibilities, guidance and Department-wide training in regards to the application of the IP which have led to wide variations in the application of the IP across and within regions. Specifically, the audit revealed inconsistent practices and unclear expectations regarding:

• The exercise of judgment when considering:
  • The unique circumstances of individual recipients (e.g. recipients judged as not benefiting from intervention despite conditions of default); and
  • Identifying the authority to intervene (e.g. events of default as provided in the IP interpreted as "triggers" vs. "indicators")
• Accountabilities, responsibilities and expectations across regions, programs and HQ related to:
  • Design effectiveness;
  • Implementation; and
• Compliance monitoring.

The audit revealed a management control framework over the application of the IP that reflects an inconsistent understanding of the distribution of roles and responsibilities across the wide range of activities that comprise the Department's intervention regime. For example, while it was clear that TPD has responsibility for the publication and distribution of the IP, it was less clear what role, if any, programs or regional operations had in ensuring the IP was designed in manner that best aligns with program objectives and/or operational considerations. The audit also noted issues related to lack of clarity regarding accountabilities for monitoring the IP.

The audit also identified that IP training is not mandatory nor is there a coordinated, national approach to training on the application of the IP. This lack of consistent and formal training has contributed to variations in the application of the IP across the Department. For example, the audit noted instances where quarterly monitoring and timely receipt and review of Remedial Management Plans (RMPs) were not being performed due, in part, to the fact that some Funding Service Officers (FSOs) were not aware of the reporting requirements.

While the scope of this Horizontal Audit did not include an assessment of the appropriateness or effectiveness of the IP, the perceived ineffectiveness of the IP was found to be impacting how it was applied within and across regions. Specifically, various levels of management and staff revealed differing views regarding the effectiveness of the IP in supporting recipients to remediate conditions of default and otherwise re-establish self-sufficiency. Among those expressing concern over the effectiveness of the IP, many cited a perceived risk that the current IP can create a disincentive for recipients to remedy the conditions that resulted in the intervention, for example, by nurturing the particular recipient's dependence on the Department; a situation contrary to the very aim of the IP, but thought to be most significant when management was considering the appointment of a third party manager. Despite the guidance provided within the IP, third party management reportedly would not be triggered by regional management if the dependency risk was thought to be high.

5.2 IP Compliance Monitoring

There is a lack of effective and efficient monitoring of compliance with IP requirements on a national or regional level.

Audit tests of recipient files at regional offices revealed a number of examples of non-compliance with the IP. Further, there are gaps in the Department's ability to monitor and effectively manage compliance with the IP at both a national and a regional level.

Compliance Monitoring at a National Level

Although the IP states that the Director, Transfer Payments is responsible for providing for the periodic review of regional compliance and implementation of the IP, the audit found that there had been only limited/ad hoc monitoring performed during the period under scope. Further, this monitoring was neither risk-based nor integrated with regional compliance monitoring. As a result, there is an increased risk that non-compliance with the IP will not be detected and remedial action taken.

In order to ensure timely detection of non-compliance with the IP, regular audits/reviews should be conducted across all regions. These audits should not be limited to specific elements, such as reporting requirements, but should be all encompassing to include items such as the appropriateness of the level of intervention applied to a recipient, recipient monitoring performed at a regional level, timeliness of receipt of required recipient reports, appropriate review of recipient reports, monitoring of Co-Management and Third-Party Management agreements as well as timely communication with recipients.

Compliance Monitoring at a Regional Level

Consistent monitoring of recipient progress allows regions to identify any significant issues that have arisen since the most recent recipient status review. Failing to perform such monitoring activities could result in an inappropriate level of intervention being applied to recipients or other issues not being detected in a timely manner.

Below are a number of monitoring-related activities assessed by the audit and the related observations:

Quarterly Monitoring

The IP stipulates that the Minister will monitor and document on a regular basis (at a minimum quarterly), the progress of the intervention mechanism. Ongoing monitoring includes, amongst other activities, ensuring that the difficulties which gave rise to the default are being addressed and that actions taken appropriately address these difficulties.

It was noted through file reviews across the 7 regions visited, that over 38% of files did not have supporting documentation to demonstrate that ongoing monitoring (at a minimum quarterly) of the recipients was being performed. Since there are no standard forms or templates for documenting monitoring activities, the audit also revealed a wide range of documented evidence (comprehensive through to none existent) supporting the completion of monitoring activities. For example, while some regions documented their detailed analysis of recipient quarterly reports, other regions indicated a lack of resources to perform a detailed review of quarterly reports. In these circumstances, an ad-hoc review was performed with a detailed analysis completed only at year-end. The audit also identified wide variances in the documentation of recipient site visits which ranged from Alberta (where trip reports were completed for recipient site visits that outlined discussion topics, key issues and action plans) to other regions where no documentation of site visits was maintained.

Remedial Management Plans

An RMP is a plan, developed by the recipient and approved by the Minister, which reflects measures to be taken by the recipient which are necessary to remedy a default under a funding arrangement. RMPs are essential to the effective monitoring of a recipient's progress toward addressing a condition of default. Quarterly reports are obtained and utilized to evaluate a recipient's status against the established RMP. In 47% of files reviewed, regions did not obtain an RMP in a timely manner and no consequences were imposed on the recipient. By not enforcing these requirements on the recipient and not holding them accountable for not fulfilling their responsibilities, a recipient does not have as much incentive to rectify the issue(s) that resulted in the default which placed them under intervention.

Tracking Recipient Status

Monitoring recipient status over a period of time is essential in determining the IP's effectiveness and in determining if alternate courses of actions are required to rectify the situation that led to the recipient's default. The audit noted that regions are not consistently maintaining information to track recipient progress. Some recipients have been under some level of intervention for a number of years. Although regional management and staff are in the best position to be knowledgeable on the various recipients' history, this information is not formally maintained and communicated at a National level. The lack of sufficient monitoring over time, and documentation of such monitoring, could result in a recipient being placed under an inappropriate level of intervention.

Quality Assurance Reviews

The IP alsostates that Regional Directors General (RDGs) are responsible for implementing quality assurance reviews to determine if planned and actual Regional processes conform to the requirements of the IP. Through interview and file reviews performed within the 7 regions visited, the audit noted that no region had performed compliance reviews on the accuracy and completeness of the application of the IP, during the period under review. As a result, regions may not detect non-compliance with the IP in a timely manner.

Recommendations:

1. The IP be reassessed with a view to clarifying policy objectives, revising policy components as needed to ensure consistency with the objectives, and providing related tools and guidance to ensure the policy is implemented as intended.

Policy revisions and related guidance to provide, at a minimum, clarity regarding:

1. IP-related roles, responsibilities, accountabilities and expectations regarding IP design, development, implementation, and monitoring across key stakeholder groups including the CFO Sector, Programs and Regional Operations (particularly FSOs).
2. When a recipient's level of intervention should be escalated or de-escalated.
3. Expectations for a risk-based and integrated strategy and approach to evaluate compliance with the IP at a National and Regional level.
4. Minimum standards of documentation related to ongoing monitoring activities, such as trip reports (notes of site-visits), minutes of meetings, decisions, action-items and responsibilities to ensure a consistent approach within and across regions.
5. Practices, including monitoring, related to the timely receipt and review of RMPs as well as the enforcement of IP requirements in cases when RMPs are not received from recipients.
6. An approach for tracking the status of recipients under intervention, tracking and documenting progress against plans at a regional level and for providing regional information to HQ for purposes of assessing overall effectiveness of the IP, and
7. A sustainable strategy and approach be established to performing regional quality assurance reviews to monitor and validate compliance with the IP.

2. In the short term, tools and practices (e.g. formal criteria, guidance and training) be developed, based on risk and current best practices, that will support consistent application of judgment to a particular recipient situation and to clarify the conditions of default as prescribed in the IP.

6.0 Observations and Recommendations - Quality Management Program

Key conclusions

A key conclusion of the Horizontal Audit is that roles, responsibilities, accountabilities and expectations regarding Quality Management have not been clearly defined. The lack of a clearly defined, Department-wide, and communicated approach to quality assurance and related responsibilities has resulted in wide variation in those practices and governance structures intended to promote and validate compliance with program and agreement Ts&Cs, policies and procedures. This has contributed to a higher risk of non-compliance with Ts&Cs, polices and procedures. It also increases the risk that personnel engaged in quality assurance activities lack the required expertise, experience or independence.

The audit also concludes that compliance reviews are not consistently performed nationally or regionally. There are three specific examples of this issue: (1) compliance reviews for Education and Social Programs were not being performed within the prescribed timeframe. (2) compliance reviews for funding arrangements with 3rd parties were not being completed and (3) CFNFAs were being rolled-over without being subject to compliance reviews.

Finally, the Horizontal Audit concludes that monitoring of recipient reporting is not consistent across regional offices. For example, of the sample of files audited, 72% did not demonstrate adequate documentation of ongoing recipient monitoring performed. It was also evident that no formal guidance or standard for the audit review process (review of a recipient's audited financial statements) had been developed or documented by HQ and disseminated to the regional offices. As a result, regional Audit Review Committees reflect varying levels of formality, capacity, process, tools and overall participation within the audited financial statement review process.

Background

A QMP is critical in supporting management's objective of ensuring compliance with funding arrangements, policies and other requirements. An overall QMP, in the context of the delivery of G&C programs for the purpose of this audit, has been broken down into two major categories of activities:

1. Quality Assurance – The objective of a Quality Assurance program is to complete independent quality assurance activities to objectively validate compliance with program/agreement terms and conditions and with applicable TBS and INAC policies and requirements (e.g. when a regional compliance unit or a HQ function conducts a review of file or other evidence to ensure that compliance requirement have been addressed).
2. Quality Control – An effective quality control program, in the context of G&C program delivery, is designed to ensure adequate management controls are in place to promote compliance with program/agreement terms and conditions and with applicable TBS and INAC policies and requirements (e.g. activities related to the documentation of files or the completion of forms that address compliance requirements associated with the funding arrangement).

The key difference between QA and QC activities is that QC activities are controls designed within the operational processes to support overall compliance while quality assurance activities are performed subsequent to the operational processes, and are performed by an independent party who is in a position to confirm overall compliance. As an example, a key quality control within the G&C process is the formal review and approval of the contribution agreement and associated terms and conditions prior to distribution to the recipient. An associated quality assurance activity includes the compliance reviews completed at the regional office level to ensure that the recipient has complied with the terms and conditions of the contribution agreement.

6.1 Roles, Responsibilities and Accountabilities

Roles, responsibilities, accountabilities and expectations regarding Quality Management have not been clearly defined.

An effective Quality Management Program requires a coordinated and integrated approach to quality control and quality assurance where all stakeholders understand their roles and responsibilities. Across the wide range of activities that comprise the delivery of G&C programs in the Department, there is a lack of clearly defined roles and responsibilities for quality management. This lack of clarity extends to stakeholders within HQ and at the regional level and includes those representing regional operations, programs and finance. Without the requisite clarity, the Department lacks an effective QMP and, as such, there is a higher risk of non-compliance with Ts&Cs, polices and procedures. Further there is an increased risk that personnel engaged in quality assurance activities lack the required expertise, experience or independence to execute their assumed quality management roles.

HQ Roles and Responsibilities - In terms of HQ involvement in quality management activities, the primary participants should be the Assistant Deputy Minister (ADM) Regional Operations, all Program ADM and the CFO (particularly the TPD). The audit found that none of these stakeholders had clear accountabilities relative to the development and operationalization of an integrated QMP on a Department-wide basis. Further, there was no articulation of how these stakeholders were intended to coordinate activities in the interest of ensuring ongoing compliance with requirements. For example, program areas at HQ were not generally seen as active participants in conducting, shaping or monitoring the quality assurance activities within the regions. As another example, the role and expectations of TPD relative to quality assurance are not consistently understood.

Regional Roles & Responsibilities - The audit noted that neither the programs (Education, Social and Capital) nor TPD have prescribed nor clearly defined QA activities required at a regional level. The audit concluded that this lack of guidance has contributed to inconsistencies in what QA activities are taking place at a regional level. Most regional offices are performing a limited number of compliance reviews on recipients' adherence to program requirements; however, the extent, nature and functional area performing these activities vary across the country. Further, these compliance reviews were not found to be risk-based. Beyond the conduct of compliance reviews, most regional offices do not have any additional QA activities. One regional office has made an attempt to develop a QA role within Corporate Services (and therefore, independent from program delivery and G&C agreement administration), with the mandate of periodic compliance reviews of program delivery activities, including compliance to the IP, Financial Administration Act (FAA) and other internal guidelines within the administration of G&C agreements; however, this role has been vacant and as a result, the QA program has not been fully implemented.

In addition to the increased risk of non-compliance, unclear roles and responsibilities relative to the QMP can lead to other risks including those related to inadequate segregation of duties (e.g. unclear responsibilities increases the risk that one individual may accept duties that are not compatible) and decision-making that is not aligned with intended authorities or required expertise/experience. The common elements of a QMP would include clear roles, responsibilities and expectations for stakeholders in respect of matters such as:

• Overall accountability for compliance (e.g. clarity regarding ownership of the QMP Program over Gs&Cs);
• Overall responsibility to ensure compliance on a day-to-day basis;
• Responsibility for monitoring and reporting the level of compliance at both the regional and national level;
• Responsibility to ensure that training, tools and guidance are sufficient and appropriate at both the regional and national level, and
• Responsibility for independent monitoring of compliance (quality assurance) at both national and regional levels and for initiating corrective action when required.

6.2 Compliance Reviews

Compliance reviews, a key existing control to ensure that recipients are complying with program Ts&Cs, are not being consistently performed.

Together with reviews of audited financial statements, ongoing monitoring, quality assurance reviews, recipient audits and other practices, compliance reviews are a critical component of QA as they help the Department assess a recipient's adherence to program and funding agreement Ts&Cs. However, compliance reviews were not being performed in accordance with requirements. Specifically, compliance review schedules as prescribed in compliance directives for Education and Social programs are not consistently achieved. It was also observed that compliance reviews for funding arrangements involving third parties were not always being completed. Finally examples were noted of CFNFAs that were being rolled-over without being subject to compliance reviews.

It was also noted that, the approach to compliance reviews was not risk based in terms of file selection and assessment procedures.

Compliance Reviews – Regional Practices and Approach to Delivery

The ability to comply with compliance review timetables as prescribed in various compliance directives varies largely from region to region. Various approaches to resourcing the compliance review function exist, e.g. a region borrowing staff from another region , outsourcing compliance reviews to outside firms such as Audit Services Canada. As some regions were not able to keep up-to-date, the compliance reviews were not being completed according to the compliance directives. This increases the risk that issues impacting the successful delivery of program outcomes may not be detected in a timely fashion.

Further, different approaches are used to identify the files to be subject to compliance review. While there are examples of a risk-based approach to compliance reviews, most conduct the reviews on a cyclical basis. In cases where resources are limited, the failure to take a risk-based approach to compliance reviews will diminish the value of the reviews as scarce resources are not efficiently allocated. For example, completing compliance reviews on a FN with a long history of accurate, reasonable and timely reporting is unlikely to yield value-added results compared to dedicating those resources to recipients with a reputation of having issues meeting the requirements of their respective agreements.

Finally, Regions regularly use their own discretion regarding the assignment of compliance roles and responsibilities to an individual or a group of individuals. A number of different models exist across the regions, including the following:

• Responsibility for compliance reviews are delegated to a QA group that is in charge of planning and executing the reviews with input from Programs and Funding Services;
• Responsibilities for compliance reviews are delegated to program staff located in the region; and
• Responsibilities for compliance review are delegated to the Funding Service Officer assigned to the particular recipient.

The third example above is a particular area of concern. Not only is the workload of FSOs incompatible with the requirements to perform compliance reviews, there are issues related to their expertise and independence. Without sufficient independence, compliance reviewers may lack the ability to objectively challenge a recipient's compliance based on agreed Ts&Cs. Regardless of who is charged with the responsibility for compliance reviews, these resources require the appropriate training, expertise and experience to conduct such reviews.

Not only has this resulted in wide variances in how compliance is monitored at a regional level across the country, this ad hoc approach also increases the risks related to inappropriate segregation of duties (e.g. compliance reviews being performed by individuals with a vested interest in the recipient's ongoing compliance) and a risk that individuals may lack the training, expertise (e.g. familiarity with agreement Ts&Cs, recipient reporting, or unique risks related to the recipient or the program) and experience to effectively complete a compliance review.

Funding Provided to Third Parties

A relatively unique issue exists in respect of instances where Departmental funding is provided to a third party (e.g. provincial government ministries) which, in turn, provides a service to a First Nation (FN). Similar to direct funding arrangements with FNs, compliance reviews for third party funding are not performed consistently or in a timely manner. A unique risk exists, however, insofar as one region indicated that compliance reviews are not performed on these agreements as they do not believe the same requirements applied as the services were provided by another government organization. While there may be less risk associated with other governments, the Departmental responsibilities remain and hence the need for compliance review should remain. To the extent there are inconsistent views as to when compliance reviews are required, there is an increased risk that specific funding arrangements may not be subject to any compliance reviews.

Compliance Reviews Performed on CFNFAs

CFNFAs are multi-year funding agreements that provide the recipient with flexibility regarding the nature and timing of expenditures within the terms and conditions of the agreement. Compliance directives for Education and Social Development programs require that CFNFAs be subject to compliance reviews only when renegotiated. CFNFAs can be rolled-over for a further 5 years (e.g. with the same terms, conditions, funding, etc.) instead of being renegotiated. If an agreement is rolled-over it would therefore not be subject to a compliance review for a further 5 years from the original arrangement date. One CFNFA had been rolled-over twice, therefore, no compliance activity had been performed on this agreement in over 10 years. Without regular review and compliance auditing activities, the Department has no means of ensuring that a recipient has met the minimum Program requirements and/or has otherwise complied with the intent behind entering into a flexible arrangement. As a result, non-compliance with program Ts&Cs may go undetected for years. The risk related to non-compliance is greater given the recent shift to the increased use of CFNFAs.

Further, the recently updated Policy on Transfer Payments and the related Directive, as issued by TB, requires that factors indicating potentially increased risk are subject to independent reviews. To the extent that CFNFAs reflect increased risk due to their duration and the flexibility offered to FNs, there is an expectation of ongoing risk-based compliance monitoring.

6.3 Quality Control Programs within the Regional Offices

Program monitoring and the audit review process (review of recipient's audited financial statements), two key elements of quality control, are inconsistently applied across regions.

The receipt and review of key recipient reports are a critical component of the monitoring of recipients for compliance with agreement terms and conditions and the status of program delivery.

Program Monitoring

There are inconsistent practices across regions in respect of both passive monitoring (e.g. receipt and review of recipient reports) and active monitoring (e.g. ongoing recipient monitoring typically performed by FSOs).

In terms of passive monitoring, delays are noted in both the receipt and review of these reports. Further, there is a general lack of documentation regarding outcomes of reviews and related follow-ups. Similarly, completion of active monitoring activities are not consistently documented in the files. This means that there is little or no evidence to support the completion of day to day monitoring activities such as site visits and meetings that support timely identification of issues related to program delivery and compliance to agreement terms and conditions. Of the sample of files audited, 72% did not demonstrate adequate documentation of ongoing monitoring. Without sufficient documentation, the regional offices cannot demonstrate that this critical monitoring activity is being consistently or appropriately performed. This limitation has been recognized by some regions and, as a result, they have taken recent steps to improve the level of documentation.

Audit Review Process

The recipient's annual audited financial statements are currently considered to be one of the most critical reports monitored by the regional offices. While delays in receiving audited financial statements occur in many cases, audited financial statements are generally provided by recipients in all regions.

Due to the importance of the information within the financial statements in the overall monitoring of the recipient to ensure delivery of program objectives, detailed review is critical. INAC's Financial Management Manual (FMM) Chapter 5.16 outlines the required analysis to be performed once the audited financial statements are received. Once the preliminary analysis is completed, the audited financial statements are required to be presented to the Audit Review Committee (ARC) as a means to ensure that the analysis completed by the FSO is complete and accurate.

As a key component of the regional QC program, the ARC is responsible for the approval of the recipient's annual audited financial statements and for recommending decisions related to intervention, as stipulated in the terms and conditions of the particular funding arrangement. No formal guidance or standard for the audit review process has been developed or documented by HQ and disseminated to the regional offices. As a result, regional ARCs reflect varying levels of formality, process, tools and overall participation within the audited financial statement review process. Examples exist of formal ARCs with specific terms of reference, composition (including formal participation of senior regional representatives and accounting expertise) and formal meeting processes (including the maintenance of minutes from each meeting) while other regional ARCs are far less formal in all respects. A common concern across nearly all regions is the lack of participation in the audit review process by program representatives. The expertise and insight that program personnel can offer is invaluable to identifying context and program-relevant criteria necessary to effectively evaluate a recipient's financial stability.

The lack of formality, inconsistent approaches to the completion of the audit review process and the limited participation of relevant stakeholders in the ARC increase the risk of inappropriate decisions related to a recipient's status. This could lead to missed indicators of potential recipient financial difficulties impacting program delivery and potential misapplication of the IP.

Recommendations:

1. An integrated and standardized Quality Management Program* be developed and operationalized.

The QMP to provide, as a minimum, clarity regarding:

1. Roles and responsibilities of key stakeholders in regions and at HQ.
2. The Departmental approach and policy that defines expectations for implementing, monitoring and sustaining a risk-based approach to compliance reviews (including programs delivered via third party agents.
3. Departmental policy/guidance regarding acceptable model(s) for ensuring timely compliance reviews taking into account the attributes of those individuals assigned with such responsibilities (including their independence from the day-to-day activities).

*Quality Management Program components to be tailored to the unique circumstances of Northern Regions.

2. Standard expectations, guidance and tools be established for the implementation of quality control activities at the regional level, including guidance regarding the monitoring, review and documentation of recipient program reports.

3. Standard requirements be defined for the review of recipient's audited financial statements at a regional level, clarifying expectations related to the formality of the ARC and representation on the ARC.

4. An audit clause be incorporated in the new CFNFAs outlining the right to conduct timely compliance reviews and program directives include CFNFAs within the requirement for audit, including stipulated frequency of these audits.

7.0 Action Plan

8.0 Annex A - Audit Criteria

Footnotes

1. Taken from AES's October 2007 Document "Grants and Contributions Audit Criteria". (return to source paragraph)