Evaluation of the Cabinet Directive on the Federal Approach to Modern Treaty Implementation

List of Acronyms

Executive Summary

A modern treaty is a nation-to-nation relationship between an Indigenous people, the federal and provincial crowns and, in some cases, a territory. Modern treaties are established to define rights of Indigenous signatories in areas such as land and resource rights, self-government, culture and language, fiscal relationships, etc. They are intended to improve the social, cultural, political, and economic well-being of the Indigenous peoples concerned. Modern treaties not only define the relationship between the Crown and Indigenous peoples, treaty rights are constitutionally protected and have the force of law that is binding on all – including federal legislation and policy. Canada is currently a signatory to 25 modern treaties, the scope of which includes thousands of obligations that the Crown must fulfill. While some of these obligations can be identified as the responsibility of a specific department or agency, others are the responsibility of many or even all departments and agencies. Accordingly, a whole-of-government approach is, in some cases, required to address these cross-cutting obligations, as well as any issues or risks that may impede their implementation.

The Cabinet Directive on the Federal Approach to Modern Treaty Implementation was adopted in 2015. The Cabinet Directive calls for a whole-of-government approach to managing Canada's modern treaty obligations and defines the roles and responsibilities of the federal government. It establishes an operational framework for the management of the Crown's modern treaty obligations and it requires the Government to enhance awareness, accountability, and oversight of modern treaty obligations. The Cabinet Directive was accompanied by a Statement of Federal Principles on Modern Treaty Implementation, which was developed in consultation with treaty partners and provides guidance on the approach to modern treaty implementation.

In keeping with the requirement to evaluate the Cabinet Directive within five years of its implementation, the Evaluation Branch of Crown-Indigenous Relations and Northern Affairs Canada (CIRNAC) led a formative evaluation. The assessment was guided by the standard lines of enquiry related to relevance, performance and the identification of early successes. Accordingly, the objective of the evaluation was to assess the degree to which: 

1. the Cabinet Directive remains relevant in the context of the Government's current priorities, roles and responsibilities;  
2. the key elements described within the Cabinet Directive that operationalize a whole-of-government approach are working effectively and efficiently to realize the objectives of the directive; and
3. the extent to which the Cabinet Directive, since its implementation in 2015, has realized early successes and results.

The scope of the evaluation was limited to the practices put in place by CIRNAC to implement the Cabinet Directive. As it was not a horizontal evaluation, its scope did not extend to the efforts and outcomes of individual departments under the Cabinet Directive. 

Within these broad areas, the evaluation was guided by a series of evaluation questions, structured according to the following main outcomes that the Cabinet Directive aims to achieve:

• Oversight and Direction
• Monitoring
• Coordination
• Assessment of Modern Treaty Implications
• Awareness and Capacity

Management Response and Action Plan

1. Introduction

1.1 Background

A modern treaty is a nation-to-nation relationship between an Indigenous people, the federal and provincial crowns and in some cases, a territory. Modern treaties acknowledge and recognize the rights of Indigenous signatories, reconciling their inherent rights with the sovereignty of the Crown. They address matters, such as: land and resource rights; environmental protection; economic development and employment; government contracting; parks and conservation areas; social and cultural enhancement; fiscal relationships; administration of justice; and self government, etc. Modern treaties are intended to improve the social, cultural, political, and economic well-being of the Indigenous peoples concerned. Constitutionally protected, they have the force of law and are binding on all - including federal legislation and policy, and have a significant place in the hierarchy of Canada's legal and constitutional framework. Stand-alone self-government agreements are not modern treaties. While Canada is currently a signatory to 25 modern treaties, many of which include self-governing provisions, Canada is also signatory to four stand-alone self-government agreements; following a whole-of-government approach, the Crown is obligated to fully and fairly implement all of these agreements.  

Figure 1: Map of Modern Treaties and Self-Government Agreements^{Footnote 2}

Text alternative for Figure 1: Map of Modern Treaties and Self-Government Agreements

Figure 1 shows the map of modern treaties and self-government agreements on the map of Canada, along with the date they were made effective.

In Newfoundland, there is the Labrador Inuit Land Claims Agreement effective from 2005.

In Quebec there is the James Bay and Northern Quebec Agreement effective 1977; the Northeastern Quebec agreement effective in 1978; and the Cree Nation Governance Agreement effective 2018.

Overlapping Newfoundland, and some of the Hudson’s Bay is the Nunavik Inuit Land Claims Agreement effective 2008.

On the border of Quebec and Hudson’s Bay, is the Eeyou Marine Region Land Claims Agreement, that was made effective in 2012.

Encompassing Nunavut is the Nunavut Land Claims Agreement, effective 1993.

In Manitoba, there is the Sioux Valley Dakota Nation Self-Government Agreement, made effective in 2014.

In British Columbia, is the Westbank First Nation Self-Government Agreement, effective 2005; the Tsawwassen First Nation Final Agreement, effective 2019; the Sechelt Indian Band Government Act effective 1986; the Tla’amin Nation Final Agreement, which was made effective in 2016; the Nisga’a Final Agreement, effective the year 2000; and lastly, the Maa-nuith First Nations Final Agreement on Vancouver Island, effective 2011.

In the Yukon, there are eleven agreements. The Kluane First Nation Final Agreement, effective 2004; the Champagne and Aishihik First Nations Final Agreement, effective since 1995; the Carcross/Tagish First Nation Final Agreement, effective 2006; the Kwanlin Dun First Nation Final Agreement, effective 2005; the Ta’an Kwach’an Council Final Agreement, effective 2002; the Little Salmon/Carmacks First Nation Final Agreement, effective 1997; the Teslin Tlingit Council Agreement, effective 1995; the Selkirk First Nation Final Agreement, effective 1997; the First Nation of Nacho Nyak Dun Final Agreement, effective 1995; the Tr'ondëk Hwëch'in Final Agreement, effective 1998; and the Vuntut Gwitchin First Nation Final Agreement, effective 1992.

Overlapping the Yukon and Northwest Territories, is the Inuvialuit Final Agreement/Western Arctic Claim, effective 1984.

In the Northwest Territories there the Gwich’in Comprehensive Land Claim Agreement, effective 1992; the Sahtu Dene and Metis Comprehensive Land Claim Agreement, effective 1994; the Déline - Sahtu Dene and Métis - Self-Government Agreement, effective 2016; and the Tlicho Agreement, effective 2005.

Under these agreements, the Crown is responsible for thousands of obligations, which can include a variety of requirements such as those related to environmental standards, capital transfers, consultation requirements, and more. The nature and number of these obligations can be direct, requiring explicit action from a department or agency, while others are contingent and are therefore triggered by other activities. Some are one-time obligations, while others are recurrent in nature. Finally, while some obligations are unique to a single department or agency, others oblige a sub-set of departments to act while others still are whole-of-government in scope.  

The Cabinet Directive on the Federal Approach to Modern Treaty Implementation (hereafter referred to as the Cabinet Directive) was established in 2015 and lays out the operational framework for the management of the Crown's modern treaty obligations, guiding federal departments and agencies in fulfilling their responsibilities. Key components of the Cabinet Directive include:

• A whole-of-government accountability framework for modern treaty implementation, including the definition of key responsibilities across all federal players;
• A Deputy Ministers' Oversight Committee (DMOC), chaired by the Deputy Minister of Crown-Indigenous Relations and Northern Affairs Canada (CIRNAC), the mandate of which is to provide executive oversight of the implementation of the Cabinet Directive, and by extension, oversight of Canada's roles and responsibilities under modern treaties;
• Modern Treaty Implementation Office (MTIO), established within CIRNAC, which provides ongoing coordination and oversight of Canada's treaty obligations and support the mandate of the DMOC; and
• A requirement that departments and agencies conduct an Assessment of Modern Treaty Implications (AMTI) on all legislative, policy, plan and program proposals to Cabinet.

Modern treaties are negotiated by CIRNAC's Treaties and Aboriginal Government Sector, which represents the Crown at the tripartite negotiating tables and which coordinates with implicated departments to ensure that commitments made in the agreements are reasonable and implementable. Once an agreement-in-principle is established, the foundations for treaty implementation are laid out in an implementation plan, the deployment of which is coordinated by the Treaty Management Division of the Implementation Sector. At this point, the treaty moves from the negotiation stage (under the coordination of the Treaties and Aboriginal Government Sector) to the implementation stage (under the coordination of Implementation Sector). During implementation, personnel from the Treaty Management Division represent the Crown on tripartite implementation committees, which become the focal point of implementation activities. While the implementation committees play an important coordination role, individual departments are accountable for implementing obligations for which they are responsible. At the same time, whole-of-government oversight and issues management is applied by the DMOC and enabled by MTIO, in accordance with the requirements as set out in the Cabinet Directive. Figure 2 illustrates these important relationships and activities:

Figure 2: Modern Treaty Lifecycle

Text alternative for Figure 2: Modern Treaty Lifecycle

Figure two showcases a chart of the modern treaty lifecycle. There are three stages of the modern treaty lifecycle.

The first stage of the cycle, negotiation, is regarding tripartite negotiations, which are meant to define ongoing rights related to land and natural resources and often, Indigenous Self-government. Negotiators and individual departments are involved at this stage of the cycle. Negotiators from Treaties and Aboriginal Government represent the federal interests in the negotiations; and individual departments provide input to the negotiating teams.

The second stage of the cycle, pre-implementation, is to lay the foundation for successful implementation, based on an agreement-in-principle, including through the development of a non-binding Implementation Plan. Negotiators from Treaties and Aboriginal Government, with input from the Implementation Sector (Treaty Management Division), establish the implementation plan.

The third and last stage of the cycle, implementation, is to implement the obligations set out in the agreement, overseeing progress and managing issues and disputes as they arise. For the implementation stage, Tripartite Implementation Committees (with Treaty Management Division as federal lead) direct and oversee the agreements’ implementation and identify, resolve or escalate issues or disputes as they arise during implementation. Individual departments implement specific obligations as required by the agreement. Departmental leadership teams oversee departmental obligations. DMOC provides whole-of-government oversight and direction. Lastly, MTIO provides whole-of-government support on issues, disputes and governance.

Negotiaton

• Negotiators from Treaties and Aboriginal Government represent the federal interests in the negotiations.
• Individual departments provide input to the negotiating teams.

Pre-Implementation

• Negotiators from Treaties and Aboriginal Government, with input from the Implementation Sector (Treaty Management Division), establish the implementation plan.

Implementation

• Tripartite Implementation Committees (with Treaty Management Division as federal lead) direct and oversee the agreements' implementation and identify, resolve or escalate issues or disputes as they arise during implementation.
• Individual departments implement specific obligations as required by the agreement.
• Departmental leadership teams oversee departmental obligations.
• DMOC provides whole-of-government oversight and direction.
• MTIO provides whole-of-government support on issues, disputes and governance.

2. Oversight and Direction

When it was established, the Cabinet Directive set out a requirement for the establishment of a Deputy Ministers' Oversight Committee, chaired by the Deputy Minister of the former Department of Aboriginal Affairs and Northern Development Canada (now chaired by the Deputy Minister of CIRNAC). The mandate of this committee was to provide executive oversight of the implementation of the Cabinet Directive, and by extension, of Canada's roles and responsibilities under modern treaties. While departments remain accountable to Parliament through the established channels of ministerial authority and accountability, the DMOC was established to provide whole-of-government oversight and direction in order to:

• Provide program and policy direction to departments in fulfilling Canada's responsibilities under modern treaties;
• Enable decision-making (and dispute resolution), as necessary, when cross-cutting issues arise that require senior executive intervention;
• Support coordination of the federal approach to broad, cross-cutting obligations;
• Oversee the monitoring, reporting and performance measurement related to the federal implementation of modern treaty obligations; and
• To allow for meetings and engagement with Indigenous and other treaty partners as appropriate^{Note de bas de page 3}.

In the context of this mandate, the evaluation sought to assess whether there is a continuing need for whole-of-government oversight and whether DMOC, in its current state, is a relevant and effective forum for achieving the stated outcomes. The evaluation concluded that there is a continuing need for whole-of-government oversight, and that a continuation of the Committee is critical to building and maintaining an open, trusted channel of dialogue between treaty holders, Indigenous leadership, and federal leadership. It is equally important to foster and grow education and awareness activities at the highest levels of government, in support of meaningful commitment to treaty implementation. However, although some early successes have been realized by DMOC, performance is not yet optimal, in part due to the challenges of monitoring results around treaty implementation, which are discussed further in this report. 

Figure 3: Summary Assessment - Oversight and Direction

Text alternative for Figure 3: Summary Assessment - Oversight and Direction

Figure 3 illustrates whether there is a continuing need for whole-of-government oversight and whether DMOC, in its current state, is a relevant and effective forum for achieving the stated outcomes. Furthermore, figure 3 highlights early successes that have been achieved for oversight and direction.

In terms of relevance, figure 3 indicates that there is a continued need for whole-of-government oversight, and that DMOC continues to be relevant for achieving stated outcomes.

On effectiveness, figure 3 signifies that although there have been some early successes realized by DMOC, performance is not yet optimal.

Lastly, there are two early successes that have been realized. The first early success listed is an increased awareness on the part of the deputy ministerial community. The second early success listed is increased engagement with modern treaty holders.

The evaluation found that there is a continuing need for whole-of-government executive oversight of the implementation of modern treaty obligations. Cross-cutting issues exist and persist, for which whole-of-government review and monitoring are needed. One example was the development of a Nunavut-specific federal contracting policy, required under Article 24 of the Nunavut Land Claim Agreement. Multiple departments were required to put in place measures to comply with this treaty obligation; however, insufficient attention had been placed on it. The intervention of the Nunavut Tunngavik Incorporated was needed, and then subsequently, whole-of-government oversight and action was required to advance on this policy.  

The understanding of federal departments in relation to modern treaty implementation is evolving, suggesting that mechanisms to oversee and foster awareness and accountability of departments on their obligations, roles and activities continue to be needed. Moreover, while awareness of the broader Indigenous and reconciliation agenda is heightened across the federal system, interviews with deputy ministers suggest that continued discrete attention on modern treaties is critical to advance departments' understanding of and commitment to the unique constitutional and legal nature of treaties.

In addition to its relevance, the evaluation also assessed the effectiveness of DMOC and its key enablers. The evaluation found that DMOC's mandate is formally and clearly documented in a terms of reference, which sets out a clear role for the Committee in providing strategic oversight of Canada's responsibilities under modern treaties, and of the implementation of the Cabinet Directive. The Terms of Reference are comprehensive and appropriately address the key elements that would enable effective committee operations, including standing agenda items that are appropriate to an oversight body and relevant in the current context such as:

• Update on implementation hot issues;
• Update on whole-of-government approach initiatives;
• Exploration of major, cross-cutting implementation issues and obligations; and
• Action items.

The mandate specifically establishes the Committee as a forum for engagement with modern treaty holders and the evaluation has noted that this function has increased in focus with considerable effort and time being devoted to listening and engaging with treaty holders. Indeed, this increased attention on listening to, and engaging with modern treaty holders has been identified as one of the early successes of the Cabinet Directive, which needs to be maintained.

Despite these successes, the Committee's current mandate does not reflect the changing nature of the Crown's approach to rights-based agreements and the evolving nature of broader governance in relation to Indigenous issues. Specifically:

• New and Different Agreements
  The advent of other rights-based agreements that are currently being negotiated under the rights and recognition tables may have whole-of-government implications and will require some form of oversight. At a minimum, as obligations evolve through new types of agreements, mechanisms may be needed to allow the Crown to understand, in a holistic fashion, the totality of their obligations. At present, it is not clear through what mechanisms the implementation of these agreements will be overseen. As accountability for the oversight of these other agreements is established, DMOC's focus should remain on modern treaties, so as not to lose focus or momentum on these matters. It is acknowledged, however, that departments may still need a mechanism to understand the full scope of their obligations under all agreements. 
• New Deputy Minister Committee on Indigenous Reconciliation
  In February 2020, the Privy Council Office established a new Deputy Minister Committee on Indigenous Reconciliation, replacing the Deputy Ministers Task Force on Reconciliation. The mandate of this committee is to manage the political and operational issues related to implementation of the Indigenous reconciliation agenda and to facilitate horizontal engagement on major initiatives. Chaired by the Deputy Clerk, this committee includes a number of DMOC members and while the full Terms of Reference are not yet available, some mandate areas are likely to overlap. As the focus and scope of this Privy Council Office-chaired Committee become more clearly articulated, the Terms of Reference for DMOC may need to be adjusted, with clear linkages between the deliberations of DMOC and those of the Deputy Minister Committee on Indigenous Reconciliation articulated. 
• Increased Calls for Independent Oversight
  In response to calls from some stakeholders, the Government is considering the establishment of more formal, independent oversight of treaty implementation efforts. Work remains ongoing at the departmental level to identify the objectives and interests of all parties towards this end, accordingly, the evaluation did not examine these efforts. However, these directions, once set, will have an impact on the nature of the whole-of-government oversight. In the context of these deliberations, it should be noted that the evaluation found that DMOC does indeed help to foster answerability of departments on treaty-related matters. While the evaluation found that DMOC's accountability mechanisms are informal – i.e., reliant on moral suasion, dialogue and influence – members have noted that this is preferable to a more adversarial relationship that may be created by a more arms-length body, which may lead to a defensive posture and may miss the opportunity to foster awareness and learning. Equally important, DMOC has provided an important forum whereby modern treaty holders can directly engage with deputy ministers on the relevant issues, contributing positively to the relationship-building and education that is needed to foster the awareness, ownership and action on the part of federal departments, all of which is envisioned in the Cabinet Directive. Ultimately, these measures help to support accountability. Given this, as CIRNAC continues to progress this work in collaboration with its stakeholders, consideration should be given to maintaining the unique benefits and positive outcomes of DMOC. 

In light of these evolutions, adjustments to DMOC's role may be needed to ensure appropriate alignment, while providing optimal coverage of matters pertaining to modern treaty implementation. 

Through interviews with DMOC members and a review of records of decision, the evaluation found that the Committee enables good, open and honest discussion and deliberation on issues that are brought before it. Members who were consulted reported an increase in awareness and overall value in the nature of the discussions. However, the evaluation identified a number of important impediments that are restricting the Committee's ability to discharge its duties related to whole-of-government oversight, the most significant of which is the very slow progress that has been made on the development and implementation of the Performance Measurement Framework (PMF) and the Modern Treaty Management Environment (MTME). These are tools that are critical to allow DMOC and departments to monitor progress on implementation. Their delayed roll-out severely restricts the information that can be provided to DMOC in support of their oversight role. Specific observations related to these tools are provided in Section 3 of this report, Monitoring, and should be promptly addressed to better enable DMOC to understand the current state of implementation and to intervene, as appropriate, on a whole-of-government basis.  

Related, while the standing agenda items, noted in the DMOC Terms of Reference, include a number of important matters related to oversight, in practice, these items are not always addressed in meetings. Moreover, the Committee focus appears to be evolving away from oversight issues to include a greater and more frequent emphasis on listening to and engaging with modern treaty holders. While positive and also directly linked to the mandate of the Committee on matters of engagement with treaty holders, there is concern that this evolution is taking away needed attention from matters of oversight, accountability and issues resolution.

Other challenges currently exist that limit the effectiveness with which DMOC is able to exercise its oversight role. Specifically:

• Few formal or consistent protocols or channels by which cross-cutting disputes and issues are escalated to DMOC
  As outlined in Section 4, Coordination, DMOC has a specific mandate to review and, as needed, address cross-cutting issues and disputes. Weaknesses in the protocols and processes related to the identification and escalation of these issues limit the ability of DMOC to engage on and deliberate over these matters. In light of the noted gaps, DMOC cannot be assured that issues in need of attention are truly filtering up to them.
• Size and Composition of DMOC
  While there is value in having a broad composition to foster awareness and learning on modern treaty issues, the larger the size of membership, the more the Committee becomes an information sharing forum, as opposed to an oversight body. Suggestions have therefore been made to focus membership on the core group of departments that have active and substantial implementation obligations.
• Need for Enhanced Meeting Discipline
  There is some opportunity to strengthen the discipline and rigor by which DMOC meetings are managed. Although meetings are held regularly and detailed records of discussion are kept, meetings lack forward planning, standing agenda items, action management / monitoring and follow up. MTIO is mandated as secretariat of DMOC and therefore plays an important role in enabling governance and oversight. The secretariat function includes leading or driving key action items, as identified by DMOC. However, the evaluation found that MTIO's role in managing action items appears to be limited. This issue is discussed in greater detail in Section 4, Coordination, where a need for better management of action items is suggested.   
• Loss of Focus as an Oversight Body
  In part because of the weaknesses noted in the previous paragraph, some members have noted DMOC has lost its focus as an oversight body. Because of limited information provided and because few cross-cutting issues come before the Committee, members are concerned that the mandate of DMOC as an oversight, challenge and advisory body is eroding, with deputies becoming increasingly unclear on their mandate in this regard. 

Both the Cabinet Directive and the DMOC Terms of Reference specify a directional role for the Committee whereby they are expected to provide "ongoing direction and guidance to departments in fulfilling federal responsibilities under modern treaties and self-government agreements and in developing federal approaches to addressing implementation issues". The evaluation found that, in practice, little policy work is done on these matters, in large measure because no policy capacity or supporting governance currently exists. Specifically:

• As part of their coordination role, MTIO is tasked with leading inter-departmental working groups and other fora to address matters of horizontal significance. In practice, this role has been played on matters, such as the development of the PMF and the MTME, but no evidence exists that they play (or have the capacity to play) a policy development or coordination role. Although the Implementation Sector has a policy function, the linkages between their work, MTIO and DMOC are not clear. 
• Unlike other whole-of-government bodies, DMOC does not have a lower-level governance structure through which policy matters can be identified, deliberated and escalated up to DMOC for their review and endorsement. The Federal Steering Committee on Section 35 rights (at the assistant deputy minister and director general level) both have a policy stream and may provide a forum for the discussion and dissemination of policy direction related to implementation matters. However, in practice, the formal connections between these bodies and DMOC are not in evidence; moreover, the evaluation identified that implementation policy matters are not getting enough attention at these fora, in part because of the prevalence of negotiation issues being addressed at the Federal Steering Committee tables. 

Faced with a continually evolving policy landscape on matters of Indigenous rights and relationships, important and complex policy questions will need to be addressed on matters related to modern treaty implementation. In light of its mandate in this area, as outlined in the Cabinet Directive, DMOC has an important role to play on these matters and will require appropriate support to play it well.   

In light of DMOC’s role in monitoring accountability for obligations under treaties, the negotiation and establishment of new treaties and rights-based agreements have the potential to implicate departments in new and different ways, affecting implementability. Interviews have indicated that the wide array of agreements that are being negotiated and signed off have the potential to create downstream risk from an implementation perspective. However, the evaluation found that no mechanisms exist, either from an operational / working level, or from a governance standpoint, to allow DMOC to gauge, discuss and prepare for these risks.

3. Monitoring

Key to ensuring the implementation of Canada's thousands of obligations under existing modern treaties are the systems and processes that are established to support the identification, tracking and reporting on the fulfillment of the many and varied obligations. Departments and oversight bodies require accurate and comprehensive information on implementation progress to intervene and take corrective action where necessary. As noted, they also require information on emerging implementation risks and challenges associated with new agreements that are coming online. Equally important is having access to information on the results being achieved by modern treaties, as this helps to inform future and ongoing negotiations and/or re-negotiations.  

At the time the Cabinet Directivewas established, there was no effective means of systematically tracking and reporting on the status of modern treaty obligations. Although the Treaty Obligations Monitoring System had been developed in 2010, a 2013 internal audit^{Footnote 4} found that no business process had been defined or implemented to ensure that information of the status of obligations was regularly updated in Treaty Obligations Monitoring System. The audit also noted that there was no proactive monitoring and reporting to decision-making bodies taking place. In the context of these findings, the Cabinet Directivemandated MTIO to work with departments to establish ongoing oversight and accountability through the development and implementation of monitoring and reporting tools, the coordination of departmental input into these tools, and the development of an annual report provided to the Minister of Crown-Indigenous Relations.

The MTME was developed to respond to functional weaknesses in the Treaty Obligations Monitoring System and to provide a more comprehensive working environment for federal departments and agencies to manage their responsibilities, and to support monitoring and reporting to DMOC. Also committed to was the requirement for a PMF to allow the Crown to measure the effects of modern treaties and to learn lessons that could then be incorporated into other negotiations and implementation efforts. 

Given the importance of such monitoring tools to enable whole-of-government accountability, the evaluation sought to determine the extent to which MTME provides a more comprehensive portrait of the Crown's modern treaty obligations and whether it represents an appropriate and user-friendly means of identifying and tracking implementation efforts and obligation gaps. The evaluation also sought to determine if common performance measures are being identified and agreed upon, in support of a PMF, and whether they are supportive of monitoring implementation efforts. 

The evaluation found that although there is a continued need for monitoring mechanisms, the current methods are not effective and are not yet well suited to support monitoring and implementation of treaty obligations at the Department or whole-of-government levels. In light of the complexity and important inter-dependencies associated with the exercise, a robust project management regime is required to support their continued development and eventual completion.

Text alternative for Figure 4: Summary Assessment – Monitoring

Figure 4 illustrates whether there is a continuing need for monitoring mechanisms, and whether the current methods are effective and well suited to support monitoring and implementation of treaty obligations at the Department or whole-of-government levels. Furthermore, figure 4 highlights early successes that have been achieved for monitoring.

In terms of relevance, figure 4 indicates that there is a continued need and relevance for monitoring mechanisms.

On effectiveness, figure 4 signifies that the current methods are not effective and are not yet well-suited to support monitoring and implementation of treaty obligations at the Department or whole-of-government levels.

Lastly, there are two early successes that have been realized for monitoring. The first early success listed is a consistent approach to identifying Canada's obligations for tracking and reporting in MTME has been established. The second early success listed is common performance outcomes have been established.

MTME was approved in 2016, with a $2 million budget and a planned two-year project duration. It was initially developed by the Consultation and Accommodation Unit within the Implementation Branch. Subsequently, under a shared governance structure, responsibility for MTME was shared between the Consultation and Accommodation Unit and MTIO. The system was released in March 2017, as part of a phased deployment. The treaty obligations that had previously existed in the Treaty Obligations Monitoring System were migrated into MTME to support departments and agencies ('departments') in recording and tracking their planned and actual implementation activities. However, by late 2017, it was evident that the accuracy and completeness of the data migrated from the Treaty Obligations Monitoring System was not sufficient for MTME. Obligations were found to be misidentified, misclassified (e.g., direct versus whole-of-government) and, in some instances, assigned to the wrong department. Furthermore, some of the 'obligations' migrated from the Treaty Obligations Monitoring System did not in fact represent obligations, but rather rights, principles or simply definitions from within agreements.

In response to these data challenges, MTIO conducted a full review of all modern treaties to identify all federal Crown obligations and to ensure that they were classified and categorized in a consistent manner. Further use of MTME by departments was therefore put on hold. In support of this 'Agreements Review' exercise, MTIO established specific guidelines^{Footnote 5} to help ensure a consistent approach to classifying obligations, including their type and frequency. Upon completion of this work, in March 2019, treaty holders and departments were asked to validate the obligations as identified and classified by MTIO, and in May 2019, they were asked to validate their assignment to departments. Approximately one-third of departments and a few Indigenous modern treaty holders responded to MTIO's validation request as of early 2020. A number of the departments that did complete a validation review found numerous identification and classification errors, including obligations that they believed were missing. As such, it is expected that the validation exercise will continue throughout 2020-2021. During this time, the foundational data within MTME will remain unreliable from a monitoring standpoint.

Concurrent with the agreement review and validation exercise, MTIO has continued to collaborate with the Consultation and Accommodation Unit to develop/enhance MTME functionality. A new release (MTME 2.0), which is expected to apply a 'user lens to the system', is currently subject to user testing. Upon its release, departments will be expected to again start using MTME to enter implementation plans and related activities, including information on one-time obligations that have been fulfilled. However, the effectiveness of the new release, in the absence of accurate and complete obligation data in MTME as previously noted, will be limited. It should also be noted that even after the release of MTME 2.0 and the validation of all obligations, additional releases are needed to enhance functionality in key areas (e.g., to allow for the capturing of key information to understand and provide context for obligations) and to ensure the system fully supports departments in fulfilling their Cabinet Directive roles and responsibilities (e.g., supporting the completion of AMTIs and to capture key information on obligations subject to issues and disputes). Management is considering a number of these features and will be implementing some on a limited basis, including the ability for departments to capture key information for fulfilling obligations, as well as reasons for delay in filling obligations, if relevant.

While MTME has the potential to be a useful tool for departments, and to support DMOC in fulfilling its oversight function, extensive delays and the need for enhanced system functionality have meant that the system is currently not meeting the needs of departments. Moreover, based on interviews with user departments, there remains a strong possibility that departmental uptake of the system will be limited once operational. Interviews with staff from various user departments suggested that resource constraints (i.e. the level of effort that will be required by some departments to enter and validate information in a timely fashion) and the development of department specific tools to track and monitor obligations in the absence of a functioning MTME could impact eventual MTME uptake by some departments. Considering this risk, once MTME is designed and implemented, CIRNAC will need to find effective ways to promote departmental uptake and adoption of the system. Not only is it important that MTME represent a useful, functional and value-added tool, available to users as soon as possible, but also that departments be/feel compelled to use the system in support of timely and transparent information sharing.

In the context of these observations, the Implementation Sector should employ a strong and robust project management regime to support the timely and quality roll-out of MTME. This would include the identification of all significant risks to the efficient and effective development and uptake of MTME, and regular and effective oversight of the project. The evaluation found little evidence of robust project and risk management in support of MTME, nor was it apparent that regular updates are provided to or requested by DMOC. Finally, in the absence of a functioning database, interim measures should be established to support regular and reliable reporting on the state of obligation implementation. This would ensure that the Crown is able to fulfill its annual reporting obligations under the Cabinet Directive.

In addition to monitoring progress on Crown obligations, the Cabinet Directive called for the development of a PMF in support of ongoing coordination and oversight of Canada's modern treaty obligations implementation. The PMF is "one of the key tools"^{Footnote 6} of the Cabinet Directive, that is intended to allow for the assessment of outcomes/impacts associated with modern treaty implementation. MTIO was tasked with its development collaboratively with Indigenous modern treaty partners by developing:   

• common objectives shared across all modern treaties, including performance indicators to measure socioeconomic outcomes in modern treaty areas; and
• where appropriate, modern-treaty-specific PMFs developed over time with interested modern treaty partners.

The evaluation focused on the work being done to develop the common objectives and their respective indicators, targets and data collection strategies. To date, some important work has been done to identify the common outcomes that can be realized from the implementation of modern treaties. This work was conducted through a working group comprised of both Crown and modern treaty holder representatives. Six common objectives have been developed with input from the Land Claims Agreements Coalition, and were approved by DMOC (Culture, Language and Heritage; Economic Development; Environment; Governance / Relationships; Land and Resources Management; and Social Development). 

These outcomes, if measured, have the potential to determine if the broader spirit and intent of treaties are being achieved. Not only would this foster accountability through enhanced reporting on the results attributable to modern treaty implementation, but it would also help to inform future negotiations or re-negotiations in situations where objectives and outcomes are not being fully realized.

However, the evaluation also noted that efforts to measure results against these outcomes are significantly behind the original schedule, which envisioned that indicators would be identified in 2017 and the first iteration of data collection would occur from 2017-2019.

Text alternative for Figure 5: Timeline of Key PMF Milestones to date

Figure 5 is a timeline of key PMF milestones to date, starting from 2015 up until the year 2020.

About midway through 2015, the first key milestone that took place was that the Cabinet directive came into effect.

The second milestone, nearing the end of 2016, the first working group meeting took place.

The third milestone, in the beginning of 2017, was a draft approach to develop PMF presented to LCAC.

The fourth milestone, in the beginning of 2018, was ongoing work by the MTIO to document common objectives, with input from LCAC.

The fifth milestone, also in the first half of 2018, was that common objectives were presented to DMOC for approval.

The last milestone that began near the end of 2018, and at the time of the report, has continued into 2020 is the establishment of next steps which include co-developing indicator for common objectives with modern treaty partners. However, the progress has been unclear.

Although outcomes exist, MTIO has not yet finalized the establishment of key performance indicators with partners. Consequently, it has yet to establish a cohesive data collection approach that would enable data collection in an efficient, effective and comprehensive way, leveraging other information that is being collected by treaty partners and the Department (an approach which would minimize duplication of effort). The development of the PMF necessitates strong collaboration with treaty partners and other government departments to be successful. The high level of inter-dependencies with these stakeholders has inherently required additional time to allow for adequate and meaningful engagement for the development of outcomes, indicators and data collection strategies. The evaluation team understands that capacity to engage modern treaty holders may be limited as a result of the various demands on their time. Finally, there appears to be insufficient capacity on the part of MTIO to drive this work forward, with a high number of resources having been redirected to the development of the MTME. Similarly, while modern treaty holders have shown interest in this work, interviews suggest that there may not be sufficient capacity to fully engage in this process. These complexities and inter-dependencies have hindered the development of the PMF.

Stakeholders identified some additional considerations that will be integral to the ongoing progress of the PMF, these include the need to:

• address outstanding data collection challenges, such as the aggregation of data for communities with small populations;
• continue to work closely with other performance measurement initiatives being led by CIRNAC to reduce inefficiencies resulting from the parallel data collection processes underway;
• ensure that the PMF includes indicators that capture both federal and treaty partner measures in achieving outcomes (i.e. indicators that monitor collaborative implementation efforts); and
• consider how the PMF will support a strengths-based approach that uses indicators, which are culturally relevant and appropriate.

As a result, the Department has not made expected progress to address the Cabinet Directive requirement for a PMF. In turn, this, along with the previously noted gaps in the MTME deployment, restricts the ability to support DMOC in its oversight role. 

4. Coordination

Over 30 departments play a role in the implementation of thousands of specific and whole-of-government obligations under existing modern treaties. For its part, CIRNAC is not only directly responsible for a significant percentage of these obligations, it is also responsible for the management of ongoing intergovernmental relationships with all modern treaty holders, as it represents Canada on the tri-partite implementation committees. Moreover, as the lead department and chair of DMOC, CIRNAC plays an important role in ensuring an appropriate whole-of-government response on matters of modern treaty implementation. CIRNAC's ability to play these roles is directly enabled by the degree to which it is equipped to coordinate across various players. 

The Cabinet Directive was established, among other things, to strengthen coordination efforts across the federal government for the implementation of modern treaties. Specifically, on matters of whole-of-government coordination, the Cabinet Directive mandated that MTIO be responsible for: 

• serving a liaison function between the implementation committees, regional and federal officials-level interdepartmental caucuses, Federal Steering Committee, and the Deputy Ministers' Oversight Committee;
• providing a secretariat function for interdepartmental committees; and
• coordinating issues management across departments.

The coordination and facilitation of issues and disputes management is particularly critical, as it allows for potential implementation problems to be promptly identified, discussed, escalated, and resolved. Without this, there is a heightened risk of eroded relationships with treaty holders and of increased legal and financial risk associated with litigation. Although each agreement has a formal dispute resolution mechanism that can be drawn upon, some issues and disputes implicate multiple departments and/or present challenges that are likely to exist or emerge in other agreements. In such instances, an appropriate whole-of-government coordination mechanism can help to ensure more timely and fulsome identification, and resolution of the issue or dispute. Considering the importance of issues management, in developing the Cabinet Directive, it was envisioned that MTIO would have key responsibilities in relation to issues management, as follows: 

• Identify and assess issues raised in the context of specific treaties and regions for cross-cutting implications, providing updates across committees and elevating major issues to the committee system; and
• Coordinate issues management efforts in relation to issues that implicate multiple departments (i.e. cross-cutting issues and disputes).

Accordingly, the evaluation sought to determine the extent to which MTIO supports, and adds value to, issues management through the development of enhanced structures and processes for the identification, coordination and resolution of cross-cutting obligations, issues and disputes. The evaluation also considered the extent to which MTIO was appropriately positioned as a neutral and objective channel through which issues can be identified and progress towards resolution.

The evaluation concluded that there is a continuing need for the coordination mechanisms outlined in the Cabinet Directive. Because of their high degree of relevance, coordination mechanisms should continue to be emphasized as a pillar in the Cabinet Directive. However, the evaluation concluded that the current approach to coordination – particularly in relation to the identification, escalation and resolution of cross-cutting issues and disputes requires a more formalized approach. Clear roles, responsibilities and pathways to decision-makers are needed to guide the liaison and information sharing activities of MTIO, the Treaty Management Division and others, without which there is increased risk that issues will not be raised in an efficient, and effective manner. Ultimately, this has the potential to lead to increased disputes.  

Text alternative for Figure 6: Summary Assessment - Coordination

Figure 6 illustrates whether there the coordination mechanisms outlined in the Cabinet Directive are still relevant, and whether the current approaches to coordination are effective. Furthermore, figure 6 highlights early successes that have been achieved for coordination.

In terms of relevance, figure 6 indicates that there is a continued need and relevance for the coordination mechanisms outlined in the Cabinet Directive.

In regards to effectiveness figure 6 signifies that the current approach to coordination needs more work.

Lastly, one early success is outlined, which is that a dispute resolution tracker has been put in place.

As previously noted, while issues and disputes can be unique to a specific agreement or department (and therefore subject to agreement-specific dispute resolution mechanisms), others can and do have cross-cutting implications affecting multiple agreements and departments. Taking into account its coordination duties, MTIO is expected to identify and assess issues raised in the context of specific treaties and regions for any cross-cutting implications that may exist. In such instances, MTIO is expected to provide updates to oversight committees (notably DMOC) on these issues, elevating 'major issues' for committee insight and direction where necessary.   

The evaluation noted that MTIO has developed and continues to evolve some of its tools to support the identification, tracking and reporting of issues and disputes to DMOC. Currently, the key tool used for gathering this information is the Dispute Resolution Tracker (DRT). MTIO developed the DRT as a means of identifying and assessing issues raised in the context of specific treaties and regions for any cross-cutting implications that may exist. This in turn is used to support DMOC in fulfilling its oversight role and its commitment to the regular tracking of modern treaty disputes.  

The evaluation found that:

• The DRT is structured to effectively describe the key elements of the dispute, including providing an explanation of the difference between an issue and a dispute; identifying the department(s) and modern treaty party(ies) involved; describing the issue or dispute and when it was raised; and summarizing internal progress on the file. The DRT, which was approved by DMOC, is updated quarterly and provided to DMOC as part of their meeting material.   
• The process for updating the DRT is somewhat effective, although some improvements are needed. As part of the updating process, MTIO reaches out to departments and their CIRNAC colleagues in the Treaty Management Division to seek updates and additions to the information in the tracker. However, the evaluation noted that the Sector does not have a common understanding of roles, responsibilities or approaches for identifying and escalating issues. While MTIO consults with the Treaty Management Division and departments, there does not appear to be a regularized, proactive process and governance through which MTIO engages with these parties to discuss and analyze the implications of any issues that may be cross-cutting. Related to this, while MTIO's process allows for the identification of new issues, the evaluation found no evidence that MTIO has a process whereby the identified issues are actually assessed for their cross-cutting implications. As such, there is little assurance that the matters being identified on the DRT are either comprehensive or, in fact, cross-cutting in nature. Indeed, interviews with DMOC members have indicated that more discussion is needed as to what constitutes a meaningful dispute or issue for escalation.    
• Finally, while the DRT is provided to DMOC members, there was little evidence that DRT information is discussed substantively at DMOC or that MTIO seeks out, or is provided with, direction from DMOC on specific matters. Rather, the DRT appears to be an awareness tool rather than a tool for capturing issues and supporting meaningful discussion. DMOC members confirmed this, noting that reasonably complete and accurate information is needed to allow members (and particularly the Chair) to know when interventions may be needed.

It should be noted that other means exist for identifying and raising issues to DMOC. For the past year, representatives from modern treaty holders have been invited (one per meeting) to present to DMOC. These presentations frequently raise challenges faced by them in implementing their agreements, including issues in relation to Crown obligations that have not been fulfilled and other related challenges. Initially, while DMOC members would make commitments to review issues that were raised in relation to their departments, there was no formal or structured follow up by MTIO, and no process whereby DMOC members were expected to report back on their findings or commitments they made. MTIO has advised that they recently established a process for recording and tracking issues raised by presenters and to follow up on commitments made to them by DMOC members. The new process and tool implemented by MTIO was not subject to review by the evaluation team as it was just recently put in place.

In addition to their role in identifying, analyzing and escalating issues and disputes, as part of their coordination role, MTIO is also expected to play a leadership and coordination role in support of their resolution. This includes working across departmental lines or with other stakeholders to identify and then coordinate solutions for noted issues or disputes. In this way, MTIO is one of many "landing spots" to which modern treaty holders may go to raise issues and concerns. To do this work successfully, MTIO requires appropriate resources and a positioning within the organizational structure that allows it to have access to the necessary decision-makers and stakeholders. The evaluation found that, at present, MTIO is not well equipped – in terms of either resources or its organizational structure and placement – to optimally fulfill these expectations.  

First, MTIO has no specific staff members or organizational entity dedicated to the role of issues management. As illustrated in Figure 7, MTIO has three separate divisions: one dedicated to the development of the MTME and PMF and reporting (monitoring); one dedicated to policy; AMTIs and training; and a third responsible for DMOC, including liaising with modern treaty holders and Land Claims Agreements Coalition (as well as providing some policy related support). While specific staff are responsible for maintaining the DRT and for following up on issues raised by modern treaty holders to DMOC, they are scattered through the organizational structure and have other, unrelated duties (mostly in relation to the office's DMOC secretariat role). If MTIO is to play a more formal role in issues management, a more structured and well-resourced organizational structure will be required to facilitate this.  

Text alternative for Figure 7: Implementation Sector Organizational Structure

Figure 7 showcases the current organizational structure of the implementation sector, organized by hierarchy.

At the highest level, is the implementation sector Assistant Deputy Minister or ADM (Ex-04).

At the next level, there is the Policy, Planning, and Coordination Branch Director General (EX-03); the Implementation Branch Director General (EX-03); and the Consultation and Accommodation Unit ADM (EX-02).

At the next level, there is the MTIO Senior Director (EX-02); the Director of Treaty Management, West (EX-01); and the Director of Treaty Management, East (EX-01).

At the next and last level, there is the Manager for DMOC/Other; the Manager of MTME/PMF; and the Manager of AMTI/Training.

Second, the evaluation surfaced some concerns in relation to MTIO's organizational placement and limited authority to compel other departments and agencies to "come to the table" on cross-cutting issues. Through its role in advising on the DMOC agenda, it has the power to escalate issues or disputes to the Committee; however, its ability to compel departments to act on these issues may be limited by its relatively low level in the bureaucracy. Moreover, the evaluation heard that the placement of MTIO within the Implementation Branch of the Sector may create a perception of a lack of neutrality relative to the specific role in the identification and escalation of issues and disputes. Although treaty holder may view MTIO differently, the Cabinet Directive did not create it as such. Instead, its role clearly states for it to perform specific whole-of-government coordination functions. As illustrated in Figure 7, MTIO is housed in the Implementation Branch. The Implementation Branch also houses the Treaty Management Directorates, which represents the federal Crown on the Tripartite Implementation Committees established for each modern treaty. For disputes raised at the Implementation Committees, the Implementation Branch can take on various roles on behalf of the Crown depending on the nature of the dispute, including coordinator or facilitator of that dispute, and is guided in all cases by terms and obligations set out in the Dispute Resolution chapter of each treaty.

Concern was raised by some treaty holders that some may be reluctant to use the MTIO channel for issues identification, preferring to go directly to DMOC instead. Perception of organizational conflict of interest due to where MTIO is positioned may work against the willingness of parties to raise disputes and issues through the MTIO channel and to have them act in a more formal issues management role. 

It should be clarified that no allegation of personal conflict of interest has been raised, nor has there been any evidence whatsoever of a lack of impartiality with respect to the coordination role and function of MTIO within the Implementation Branch. This observation by treaty groups may be premised on the broader notion – long held by some treaty groups – that the Government cannot be both the 'implementer' of treaties and the overseer of its own performance at the same time.

In summary, if MTIO is to assume a more active role in the coordination and management of cross-cutting issues and disputes, barriers, such as resources, will need to be addressed, along with the establishment of more formal processes to identify, manage and escalate the issues. Furthermore, roles and responsibilities and processes may also need to be clarified with a view to establishing a clear and regularized process by which MTIO, the Treaty Management Division and departments work together to identify, analyze and then address cross-cutting issues.

5. Assessment of Modern Treaty Implications

Because of the breadth of modern treaty obligations, there is a high potential that government actions in various areas may intersect with the treaty rights of modern treaty holders. Accordingly, under the Cabinet Directive, departments are obligated to consider the implications of modern treaties as they develop new legislation, policy, plans, and program, proposals to Cabinet, and to attest to the compliance of the proposals with the legal obligations contained in modern treaties. To this end, departments and agencies must complete an AMTI analysis, which poses standard questions of departments and, in so doing, compels them to consider the manner in which legislation, policy or program proposals will implicate, or be implicated by, their modern treaty obligations. In this way, the goal is to foster greater awareness of and responsiveness to treaty obligations. While mandatory for all Memoranda to Cabinet, a separate optional assessment checklist has been developed for Treasury Board Submissions. In addition to developing the AMTI tool, the Implementation Sector also provides training to departments to support its use.  

In light of the importance of this tool in fostering awareness and responsiveness of federal programs and policies to modern treaty obligations, the evaluation sought to determine if there was a continued need for departments and agencies to complete AMTIs and the extent to which the AMTI process supports departments in considering their modern treaty obligations in the development and updating of policies, plans and programs (i.e. compliance, alignment).

Text alternative for Figure 8: Summary Assessment - AMTI

Figure 8 illustrates whether there is a continued need and relevance for departments and agencies to complete AMTIs and the extent to which the AMTI process is effective in supporting departments in considering their modern treaty obligations in the development and updating of policies, plans and programs (i.e. compliance, alignment). Furthermore, figure 8 highlights early successes for AMTI.

In terms of relevance, figure 8 indicates that there is a continued need for departments and agencies to complete AMTIs, and the use of AMTIs continues to be relevant.

In regards to effectiveness, figure 8 indicates that the AMTI is generally effective but there continues to be room for improvement.

Lastly, two early successes have been highlighted for AMTI. One being the widespread roll-out of the tool with training. The second one being that individual departments have been adapting the tool to meet their needs.

The evaluation noted that departments and agencies find the AMTI tool helpful as a means to work through the implications of their policy and program proposals from a modern treaty perspective. Although the lack of a functioning data base of obligations makes it challenging to confirm the true scope of the departments' obligations, departments that were surveyed found it to be a useful tool in posing necessary questions and therefore prompting key considerations.  

The evaluation did not assess the quality of the AMTIs submitted to CIRNAC but did note that no formal processes exist within the Department to periodically review the quality of the departmental submissions. Lessons learned from the Government's Gender-Based Analysis Plus experience indicate that the existence of a mandatory quality assurance function helps to ensure the assessment is done consistently and appropriately. This type of periodic review would be best conducted by MTIO with support from the Privy Council Office and Treasury Board Secretariat.

A stronger quality assurance regime is particularly important to address the concerns noted by modern treaty holders that AMTIs are not being completed adequately. Specifically, both modern treaty and federal partners raised concerns regarding the current engagement process when conducting an AMTI. Treaty partners stated that there is currently little to no engagement occurring with modern treaty holders, leading to concerns that the process is not transparent despite potentially assessing and making decisions related to their treaty rights (i.e., departments may be able to simply "go through the motions"). Federal partners identified a general lack of guidance for the AMTI engagement process as a challenge, particularly in relation to how it relates to other engagement and consultation triggers, and the secret nature of Cabinet documents.

While the evaluation was not able to validate these concerns in light of the scope of the review, it did note that an important barrier to consultation with modern treaty holders is the secrecy associated with the tool, once completed. Tied as they are to Cabinet documents, the completed AMTIs are considered to be Cabinet Confidence, and are therefore secret documents that cannot be shared. As well, some departments suggested that they felt their consultation duties were being discharged in other ways, through the established duty to consult mechanisms; accordingly, additional consultation was not felt to be needed. In summary, additional effort appears to be needed to confirm and communicate expectations from an engagement perspective.  

The evaluation also found that, perhaps as a reflection of the demand for this tool, departments are not only adopting it, but also adapting it to meet their unique needs. Improvements are being made at the departmental level; however, no formal mechanisms exist to harvest and mainstream these department-initiated improvements for more widespread benefits. 

Although the federal government considers the AMTI process to be is generally effective, there does not appear to be a mechanism to ensure that information identified in the assessment is acted upon and implemented at the program design / re-design stage. While this would be the responsibility of the individual departments, (with potential support provided from MTIO in the form of best practices, tools, frameworks, etc.), enhanced quality assurance over the process could be designed to periodically assess the degree to which noted implications are, indeed, being addressed and responded to through the program design or policy implementation efforts.   

Finally, the evaluation found that there is a high and increasing demand for training in support of the AMTI process, with departments and agencies seeking advice, training and guidance that is targeted to departmental, regional and/or program-specific matters. Although MTIO delivers training, this was not always well understood by departments who were surveyed as part of this evaluation. Therefore, more effort to convey training and development options may be needed, as well as an expanded suite of training and tools, in order to meet growing and different needs of departments and agencies. 

6. Awareness and Capacity

Under the Cabinet Directive, a commitment to awareness and capacity-building was established to address noted knowledge gaps across the federal family on matters of modern treaty implementation. MTIO was charged with the development of training and tools for federal departments and agencies aimed at fostering general awareness, capacity-building for functional experts and enhanced knowledge of cross-cutting issues. In light of its importance, the evaluation sought to assess the relevance and effectiveness of the training and awareness methods put in place by CIRNAC.

Text alternative for Figure 9: Summary Assessment – Awareness and Capacity

Figure 9 illustrates the relevance and effectiveness of the training and awareness methods put in place by CIRNAC, in response to the commitment to awareness and capacity-building set under the government directive to address noted knowledge gaps across the federal family on matters of modern treaty implementation. Furthermore, figure 9 highlights early successes for awareness and capacity-building.

In terms of relevance, figure 9 indicates that there is a continued need and relevance for the training delivered on modern treaties.

In regards to effectiveness figure 9 indicates that the training is generally effective but opportunities exist to enhance the content and expand its delivery.

Lastly, an early success that was found, is that the training has been designed and delivered collaboratively, and is viewed positively by attendees.

CIRNAC has developed relevant training in collaboration with the Land Claims Agreements Coalition. Training sessions were originally delivered by MTIO but have been delivered it in partnership with the Canada School of Public Service since 2019-2020. Training is either one or two days in duration, depending on the level of detail desired by the participant and includes engagement with a modern treaty holder representative who speaks at each course offering. To date, 16 sessions have been delivered in partnership with Canada School of Public Service (eight of each day of the course). With support from MTIO, additional online training courses have been developed by modern treaty holders, whether the Land Claims Agreements Coalition or individual modern treaty holders, objectives of which focus on educating participants on their respective agreement. In addition to their training material, MTIO houses and maintains modern treaty tools at the MTIO Resource Centre, which is accessible to all departments and agencies. Moreover, MTIO provides ad hoc advice and support to departments and agencies on specific questions and issues as they may arise. 

Material has consistently been assessed positively by attendees, with an average course rating of 4.3 out of 5. Further, attendees who self-rated their knowledge of modern treaties in the 1-3-point range (out of 5) prior to attending the course indicated that they would rate their knowledge as approximately 2 points higher upon course completion^{Footnote 7}. Demand for training offered by MTIO and Canada School of Public Service remains high, with more training needed in French and outside the National Capital Region. Training is generally relevant and in line with current and emerging government directions, although more information is needed on the various elements of the Government's directions and evolving landscape relative to Indigenous rights and reconciliation.

Moreover, modern treaty holders have identified that there remains limited knowledge/understanding of treaty specific contexts, rights and obligations with many having noted that they spend significant effort educating federal officials when they are engaged for an initiative. As a result, a continued investment in training is required. 

In this context, additional opportunities exist to expand and enhance the training. Specifically:

• The material could be expanded to address the constitutional status of modern treaties, and their relation to departmental authority, with a view to emphasizing that treaties are constitutionally protected and represent a force of law that is binding on all, including federal legislation, programs and policy;
• Specific reference could be added on the new recognition and rights tables that are occurring, and the implications for the broader treaty landscape;
• More specialized training appears to be needed in key areas (e.g., on cross-cutting issues, region-specific matters, etc.); and
• The reach of the training could be expanded to include departmental officials who do not regularly deal with modern treaties, but for whom general awareness would be beneficial. 

Finally, the evaluation found that the bulk of the efforts to develop awareness in the federal government are training-based in nature. While this training has been positively received and is in high demand, other federal departments noted that it would be beneficial for MTIO to play a more proactive role, particularly as it relates to providing tools and guidance on challenging, cross-cutting obligations. Many opportunities to advance beyond formal training to other forms of awareness-generation and capacity building exist, including:

• An enhanced client service approach on the part of MTIO to respond to and support departments through advice and support;
• The development of case studies and simulations to allow departments to walk through challenging cross-cutting issues that would be typical for their business;
• On the job learning on key tools, such as MTME; and
• MTIO working with departments to tailor training tools to their needs and delivering a "train the trainer" program to enable departments to lead and implement their own training.

7. Conclusions and Recommendations

In the face of continuing departmental and cross-cutting obligations, and in the context of expanding and evolving relationships and agreements with Canada's Indigenous peoples, there is a continued need for the Cabinet Directiveand for the key components established by CIRNAC to enable its implementation. Accordingly, the evaluation concluded that the Cabinet Directive and its key elements are relevant and should remain in force. There is a continued need for whole-of-government oversight and dialogue to foster understanding and awareness across departments, which in turn helps to strengthen federal accountability for their obligations under modern treaties.  

As departments' understanding and awareness of the whole-of-government approach evolves, the need for oversight continues. The oversight approach used by DMOC is appropriate and relevant as a means of fostering social and organizational change in a whole-of-government context, where education, awareness and suasion are powerful forces. While more formal and independent oversight mechanisms may yield benefits in terms of enhanced accountability, caution should be taken to ensure that the pursuit of this approach does not shift the relationship into an overly adversarial one, as this would undermine the early successes achieved by DMOC in terms of opening up a trusted channel with modern treaty holders.

There is also a continuing need for monitoring and coordination mechanisms, as they are critical to support the whole-of-government actions and accountabilities that are broadly needed for modern treaties to be implemented effectively and consistently. The AMTI process and other educational / awareness mechanisms are relevant, valuable and help, together with the benefits brought by DMOC, to foster increased understanding and commitment at different levels of the federal government. 

Prompt attention should be placed on the development of monitoring mechanisms which, once in place, have the potential to greatly support oversight, accountability, learning, and course correction. These and other improvements would support and sustain the early successes that have been noted in this evaluation, the most notable of which has been the progress made on fostering dialogue and understanding across the federal system. Although more is needed to sustain and advance these successes, the following recommendations have been made to support the achievement of the results associated with the Cabinet Directive. 

It is recommended that CIRNAC: 

1. In the context of the new Deputy Ministers' Committee on Indigenous Reconciliation, as well as  the evolving discussions on the introduction of independent oversight mechanisms, review and adjust the DMOC Terms of Reference to ensure that:
   1. they align with and appropriately inform the Deputy Minister Committee on matters related to modern treaty implementation;
   2. the DMOC continues to provide a dedicated, deputy minister-level forum on matters of modern treaty implementation through which deputy ministers can address, discuss and answer to colleagues on whole-of-government / systemic modern treaty issues;
   3. they provide for an appropriate balance between oversight and engagement opportunities; and
   4. the DMOC is specifically mandated with examining and overseeing emerging risks related to implementation of existing and new modern treaties.
2. Ensure that the policy function within the Implementation Sector is appropriately equipped and positioned to support DMOC in their directional role under the Cabinet Directive. This should include:
   1. ensuring it is appropriately resourced and able to provide insight, foresight and other policy support that may be needed to set directions on matters related to modern treaty implementation;
   2. the establishment of a process by which policy priorities are set annually by DMOC to guide policy efforts of the Sector; and
   3. the establishment of appropriate governance structures and processes that enable discussion and debate on implementation policy at the level of senior officials (e.g. assistant deputy minister, director general), leading up to and supporting DMOC as a policy body. Reference these new governance structures in the Cabinet Directive.
3. Make the completion of the Modern Treaty Management Environment a priority, putting in place robust project management practices to ensure that key tasks are planned and executed. This would include, but not be limited to:
   1. the maintenance of a detailed work plan that is kept current;
   2. transparent and regular consideration of project risks, issues and challenges, including the tracking and management of mitigation measures; 
   3. regular involvement and appropriate input from stakeholders (other government departments and, if possible, modern treaty holders, perhaps focusing on those that are most implicated); and
   4. regular oversight and reporting to the DMOC on progress.
   Until regular tools exist for reporting, implement interim measures to allow for annual reporting on modern treaty implementation.
4. Make the completion of the Performance Measurement Framework (PMF) a priority and develop a fulsome data collection strategy within a reasonable period of time. In doing so, MTIO should also:
   1. put in place robust project management practices, in alignment with Recommendation 5;
   2. continue to develop the PMF in a collaborative manner, including the development of indicators for both modern treaty holders and the Crown, and that are culturally relevant;
   3. continue to work with other data collection activities underway in CIRNAC to minimize duplication of effort, streamline the data collection process and jointly address data collection challenges; and
   4. ensure that data collection supports the needs and capacity of modern treaty holders.
5. Clarify and document processes for the clear and consistent identification, escalation and resolution of cross-cutting issues and disputes, including:
   1. documenting roles and responsibilities of key players and establishing efficient processes by which all parties are expected to share information on the state of cross-cutting issues with one another; and
   2. formalizing the process by which the MTIO examines the issues and disputes noted for their cross-cutting implications, including establishing the threshold by which an issue becomes "major" and is therefore escalated to DMOC. 
6. Strengthen the mechanisms by which MTIO, departments and agencies can be held accountable to DMOC for the commitments made in response to the Committee's direction and requests. This could include the introduction and regular use of an action log or other similar mechanisms. 
7. Establish a quality assurance function within the MTIO that is responsible for the periodic assessment of departmental/agency compliance with the AMTI process. Through this or other means, identify, harvest and share lessons learned in support of continuous improvement. Update the Cabinet Directive to reflect this as a role of MTIO.
8. In collaboration with modern treaty holders, clarify and document expectations for consultations with modern treaty holders when conducting an AMTI, including how AMTI consultations relate to other triggers for engagement and to the principles of Cabinet secrecy.
9. MTIO should work with the Canada School of Public Service, departments, agencies, and modern treaty holders to enhance the support provided with a view to strengthening awareness and capacity across the federal government. This should include:
   1. developing a more proactive and client-centric delivery model within MTIO, including enhanced collaboration with federal partners to better understand and then respond to their awareness and capacity needs; and
   2. increasing the depth and frequency of learning opportunities by developing alternative mechanisms for sharing of information and best practices to ensure that federal employees stay abreast of emerging issues and developments on matters pertaining to modern treaties.

Appendix A: Evaluation Framework

Appendix B: Risks Associated with the Cabinet Directive

The following risks, used to help calibrate the evaluation, are those that the Cabinet Directive is inherently exposed to.