Audit of Performance Measurement

Background

The Audit of Performance Measurement was included in Indigenous and Northern Affairs Canada's ("INAC" or "the Department") 2016-2017 to 2018-2019 Risk-Based Audit Plan, approved by the Deputy Minister on March 4, 2016. The audit was identified as a high priority because of the importance of monitoring and assessing the results of departmental programs and due to the significant interest and concern across government in this area.

The Mandate Letters to the Ministers of the Government of Canada identify performance measurement as a priority and establish an expectation "to report regularly on your (departmental) progress toward fulfilling our commitments" made and "to help develop effective measures that assess the impact of the organizations for which you (Ministers) are answerable".^{Footnote 1}

Performance measurement activities refer to those activities undertaken to support monitoring, measuring, evaluating and reporting on the performance of the Department's programs and services. Like all departments, INAC is managing two major performance measurement transitions: the implementation of the Treasury Board (TB) Policy on Results and the implementation of "Deliverology", which is a collection of results-based management practices to fulfill the Government of Canada's commitment to deliver results in public services.

Audit Objective and Scope

The objective of the audit was to assess the Department's readiness and capacity to implement the changes required under the TB Policy on Results as they relate to performance measurement and to identify relevant performance measurement considerations.

The scope of the audit included assessments of the adequacy and effectiveness of INAC's performance measurement system with a focus on the risk areas identified during the planning phase. The audit focused on the period from March 31, 2015 to August 1, 2016, however included the review of some earlier documention.

The audit scope included the assessment of compliance with program performance measurement and excluded performance measurement related to internal service delivery (e.g., finance, human resources, etc.). Specifically, employee performance management, time and expense performance, or areas of performance not directly related to measuring INAC's delivery of programs and services were not included within the scope of this audit.

Statement of Conformance

The audit conforms to the Internal Auditing Standards for the Government of Canada, as supported by the results of the quality assurance and improvement program.

Positive Observations

The audit identified a number of positive observations related to INAC's performance measurement, grouped by thematic area:

Conclusion

The audit found that the Department has initiated relevant actions to support its readiness and capacity to implement the changes required under the new Policy on Results. To support the transition to the Policy on Results, opportunities to aid its implementation have been identified. These include: clearer, better communicated strategic direction; data governance consistent with requirements of the new policy; expanded understanding of data needs for planning in Indigenous communities that can be used for performance reporting; and, more consistent use of performance information across the Department to achieve strategic objectives.

Considerations

The audit team identified key areas where improvements are required in order to support departmental readiness and capacity to implement the Policy on Results and the Government's Results Agenda. Considerations are outlined below:

"Deliverology" and the Government's Indigenous Agenda

The Government of Canada indicated through the Cabinet Committee on Agenda, Results and Communications – the Committee responsible for developing the Government's forward agenda, tracking progress, and managing strategic communications – that departments will be responsible for implementing the core tenets of "Deliverology". Pioneered in the UK public sector, "Deliverology" is a performance measurement approach that involves best-in-class performance measurement practices to drive policy outcomes by focusing on key activities most likely to have the greatest impact.^{Footnote 4}

Additionally, the Government of Canada has recently indicated through all Mandate Letters that performance measurement will be an important priority. The Mandate Letter to the Minister of INAC stipulates that "…our work will be informed by performance measurement, evidence, and feedback from Canadians. We will direct our resources to those initiatives that are having the greatest, positive impact on the lives of Canadians, and that will allow us to meet our commitments to them. I expect you to report regularly on our progress toward fulfilling our commitments and to help develop effective measures that assess the impact of the organizations for which you are answerable".^{Footnote 5} The Government has also made Ministerial Mandate Letters accessible to the public "for the first time in our country's history"^{Footnote 6}.

Key Performance Measurement-related Policy Changes

The TB 2010 Policy on Management, Resources and Results Structures (MRRS) was designed to ensure that the government and Parliament have integrated financial and non-financial program performance information to support allocation and reallocation decisions in individual departments and across the government. The Policy on MRRS required that each department and agency put in place a common framework for the identification of programs and for the collection, use, and reporting of financial and non-financial information relative to those programs. This is further supported by the TB Policy on Evaluation, which was designed to ensure that credible, timely and neutral information on the ongoing relevance and performance of direct program spending is available and used to support evidence-based decision making.

TB released the Policy on Results on July 1, 2016, which replaced the Policy on MRRS, and the Policy on Evaluation. Transition to full implementation of the new policy is to be completed in 2017-2018. During the transition period, departments are expected to continue to support requirements under the Policy on MRRS and the Policy on Evaluation.

Relevant similarities between the Policy on MRRS and the Policy on Results

The Policy on MRRS and the Policy on Results are broad management frameworks designed to enable informed decisions about resource allocation and program delivery. Evidence-based performance measurement principles, financial and non-financial data utilization, horizontality (initiatives that cut across programs and departments), program design and management flexibility, and achievement of results for Canadians through consistent monitoring and reporting are attributes of both policies.

Relevant differences between the Policy on MRRS and the Policy on Results

The objective of the Policy on Results is to ensure that, at a broader level, government departments are focused on not just short-term priorities but longer-term mandates. The key differences between the two policies as they relate to performance measurement are defined in the table below.

The Policy on Results does cover a broader spectrum than the Policy on MRRS. For example, it defines broader roles and responsibilities for: the Deputy Head; the Performance Measurement and Evaluation Committee (formerly a Departmental Evaluation Committee); program officials, and calls for the creation of a Head of Performance Measurement.

The Policy on Results and application of "Deliverology" to the Canadian federal context contains overlapping goals and approaches. In consideration of these factors, the Audit of Performance Measurement describes the current state of performance measurement, taking into consideration the comprehensive assessment of the state of performance measurement conducted and reported in the Department's 2013 Annual Report on the State of Performance Measurement, and gaps relative to the anticipated requirements under the Policy on Results and "Deliverology".

2.1 Audit Objective

The objective of the audit was to assess the Department's readiness and capacity to implement the changes required under the TB Policy on Results as they relate to performance measurement and to identify relevant performance measurement considerations.

2.2 Audit Scope

The scope of the audit included assessments of the adequacy and effectiveness of INAC's performance measurement system with a focus on the risk areas identified during the planning phase. The audit focused on the period from March 31, 2015 to August 1, 2016 and included the review of some earlier documentation.

The audit scope included the assessment of compliance with program performance measurement and excluded performance measurement related to internal service delivery (e.g., finance, human resources, etc.).

5.1 Leadership and Strategic Direction

The audit expected to find evidence that senior management actively supports a performance measurement culture through performance measurement-related communications, routine discussion of performance measurement-related information in senior committees, clear prioritization of performance measurement, and a strategy for implementing performance measurement-related changes defined in the Policy on Results and expectations under "Deliverology".

Senior Management has taken steps to support a transition to the Policy on Results and "Deliverology", recognizing the transition presents a unique opportunity for INAC to improve performance measurement effectiveness and to support a performance measurement culture in the Department.

The Assistant Deputy Minister (ADM) Steering Committee on Results and a Director General-level Delivery and Results Committee were established to support implementation of the Policy on Results and the Government's Results Agenda. The audit noted that these senior level committees are working toward developing the core requirements of the Policy on Results. For example, INAC has developed a DRF, Program Inventory, Results Charter and Mini-Charters (expectations under "Deliverology"). The positions of Chief Results and Delivery Officer and Chief Data Officer have been designated.

Leadership challenges related to communications and prioritization of performance measurement were identified. Audit work indicated that performance measurement communications related to program results were infrequent and that support for results-based management in the Department is inconsistent. A review of some committee meeting minutes for the period in question found that performance measurement-related discussions and action items focused on program risk monitoring, updates to the MAF, and updates on specific initiatives. In addition, interviews suggested prioritization of performance measurement activities is unclear.

"Deliverology" and to a lesser extent the Policy on Results are viewed by those interviewed as an opportunity to develop a stronger culture of results-based management in the Department. However, a change management strategy that defines and communicates the transition journey (timing, sequencing, dependencies, milestones, and investment levels of human and financial resources) to full implementation of the Policy on Results and "Deliverology" has yet to be developed, as the new DRF and program inventory were still in progress at the time of the audit. It was observed that the Department has put into motion a plan to implement the Policy on Results, and that key success factors were identified and include:

• Leadership, collaboration and accountabilities;
• Communication and engagement;
• Capacities and tools; and
• External engagement with Indigenous peoples.

5.2 Clear Accountability

The audit expected to find that accountabilities and roles and responsibilities related to performance measurement are clearly defined and well understood by staff.

INAC has worked in recent years to improve the governance of performance measurement. The Department has rationalized departmental program authorities, refined its PMF, and worked to clarify program-level PM Strategies, which included clarification of roles and responsibilities. The Department has developed a Policy on Performance Measurement Strategies that outlines the roles and responsibilities for performance measurement across the Department. Although out of scope for this audit, interviews suggest that performance measurement roles and responsibilities have been reinforced by the business planning process and quarterly reporting.

The audit also reviewed a selection of PM Strategies to assess the adequacy of descriptions of roles and responsibilities. Although program governance and stakeholders sections were consistent features of the PM Strategies reviewed, roles and responsibilities defined in the performance measurement matrix varied; some PM Strategies identify a position or number of positions and some identify a Directorate or Branch as responsible. While differences across PM Strategies related to governance were identified, they reasonably reflect program realities and are consistent with the flexibility afforded to program officers to develop strategies that suit the realities on the ground.

The Government's Indigenous agenda and ongoing work in and with Indigenous communities envisions an accountability structure grounded in a renewed nation-to-nation relationship. Towards this end, the Assembly of First Nations is leading a transformation to a new Indigenous Investment Management Framework, which will support public reporting on community, nation and population development as opposed to program-related outcomes. The Department is committed to working with communities to support this transition.

Although the audit found roles and responsibilities for program results are generally well defined and understood by staff, senior management recognizes that results are likely to become increasingly the responsibility of Indigenous communities. They noted that the impact of the process of reconciliation with Indigenous communities on departmental accountability for results and the implementation of the Policy on Results will need to be clarified.

5.3 Client-focused Service

The audit expected to find evidence of sufficient departmental capacity to execute performance measurement activities including knowledge of resources and available staff as well as evidence that the Department considered and incorporated Indigenous community needs into its performance measurement approach.

Interviews found that knowledge and understanding of performance measurement principles across the Department have improved as a result of work to expand coverage of program-level PM Strategies. Access to Senior Strategic Outcome Advisors, Policy and Strategic Direction results-based management specialists embedded in sectors to provide support and advice to senior management and support from Evaluation, Performance Measurement and Review Branch staff in the establishment and updating of PM Strategies were noted as useful enablers.

Capacity challenges underscored in interviews related less to available knowledge and ability of staff to analyze and report on program performance and more to staff availability from a timing perspective to conduct performance measurement activities. Audit work indicated limited performance measurement capacity including any initiatives to respond to changing/evolving needs (either from the Department or from Indigenous communities) as a result of operational and program administration requirements.

Consideration and incorporation of community needs into the performance measurement process was also assessed. In response to a series of independent enquiries, the Department has sought to reduce reporting requirements placed on Indigenous communities while continuing to provide Parliament and Canadians with information sufficient to assess program performance.^{Footnote 8} The Department has initiated consultations and pilot programs with Indigenous communities to support streamlined data collection and reporting obligations. The Simplified Reporting Initiative^{Footnote 9} has identified community needs-related challenges that future pilot projects aim to address, and which consider Indigenous reporting capacity and relevance of reporting requirements for communities. The goals of the Simplified Reporting Initiative include:

• Simplify reporting through consultation with Indigenous communities, leveraging best practices and working with programs, regions and internal services to reduce reporting requirements;
• Increase reporting flexibility through web-based solutions and predictable reporting schedule; and,
• Improve data sharing with Indigenous communities through web-based solutions.

In addition to the Simplified Reporting Initiative, INAC has also completed a Data Collection Review and Approval initiative and has successfully reduced reporting requirements contained in Data Collection Instruments (DCIs), which is described in the following section.

Interviews indicated that some programs are developing engagement strategies. However, there is no oversight body to provide direction to ensure that these engagement strategies are well developed and to facilitate collaboration among sectors.

5.4 Results and Performance

The audit expected to find that INAC had established quality and credible performance information, that it used performance information to make decisions, and that performance is monitored and supported by established systems and processes.