Archived - Evaluation of the Comprehensive Community-based Planning Pilot Project in Saskatchewan

2.1 Evaluation Scope and Timing

The evaluation examined the Comprehensive Community-based Planning pilot project activities from its start in 2005-2006 through to its completion in 2010-2011. Research for the evaluation was conducted between February and May 2011.

2.2 Evaluation Issues and Questions

The evaluation focused on the following issues: (The full evaluation matrix can be found in Appendix B.)

• Relevance
  • Continued Need
    Does comprehensive community-based planning address a demonstrable need in Aboriginal communities?
    
    
  • Alignment with Government Priorities
    Are the objectives of comprehensive community-based planning consistent with departmental and government-wide priorities? Is it aligned with federal roles and responsibilities?
    
    
  • Alignment with Federal Roles and Responsibilities
    Does the Comprehensive Community-based Planning pilot project duplicate or overlap with other departments or jurisdictions?
    
    
• Performance
  • Design
    To what extent were the project objectives/outcomes clear to all major stakeholders and shared?
    To what extent was the project design appropriate?
    To what extent was the governance structure appropriate?
    
    
  • Delivery
    Were there sufficient and appropriate resources and support to implement the project?
    Was there sufficient flexibility to adapt to changing conditions/local contexts (language and culture)?
    
    
  • Effectiveness
    What did the project achieve?
    To what extent is there increased awareness (on the part of pilot communities, AANDC Saskatchewan region and tribal councils) of the importance of a comprehensive community-based plan?
    To what extent have individual and community stakeholders increased their capabilities regarding comprehensive community-based planning?
    To what extent are the community plans being used by communities and AANDC Saskatchewan region?
    
    
  • Sustainability
    To what extent will the benefits of comprehensive community-based planning continue after the pilot projects end?
    Have the pilots had unexpected outcomes, positive or negative?
    
    
  • Economy and Efficiency
    Are the most appropriate and efficient means being used to develop community plans?
    Are there alternative models that could improve comprehensive community-based planning or reduce costs?
    
    
  • Lessons
    What mechanisms are in place to capture lessons and facilitate learning from the pilot initiatives?
    Are there lessons or alternatives that have emerged that could contribute to improved community planning?

2.3 Evaluation Methodology

2.3.1 Data Sources

The evaluation findings and conclusions are based on the analysis and triangulation of the following lines of evidence:

Literature Review:

The evaluators conducted a review of relevant regional, national and international literature related to community-based planning.

Document review:

The evaluators reviewed project documentation, including management information, contracts, project reporting, newsletters, budget information, community plans, design-build project booklets, and other related information.

Key informant interviews:

The evaluators conducted interviews with 23 key informants, including:

• Representatives of four pilot communities (6)
  
  
• Cities and Environment Unit (3)
  
  
• Tribal councils (3)
  
  
• AANDC staff in Saskatchewan region, British Columbia region and Headquarters (10)
  
  
• Health Canada representative (1)

Case Studies:

The evaluators visited five of the pilot communities over two weeks in early March 2011 to collect detailed qualitative data on the project's outcomes. In each of the communities, the evaluators conducted interviews and/or focus groups with the Plan Champion, Community Contact, Planning Work Group members, and community leadership. All five case studies incorporated information from the document review, including the community plan, project reporting and community profile data. The case studies examined best practices, lessons, key successes and how they were achieved, as well as factors that were important in facilitating or hampering success.

2.3.2 Considerations, Strengths and Limitations

There were few limitations to the evaluation. Evaluators received excellent cooperation from project stakeholders and the pilot communities, with representatives of nine of the eleven pilot communities participating in the evaluation. However, most of the communities selected to participate in the pilot project had good financial and human resource capacity, a characteristic that may limit the applicability of findings from this evaluation to similar communities, i.e. the findings cannot necessarily be generalized to communities with lower levels of financial and human resource capacity.

Since the project lacked a logic model, the evaluators created one and validated it with three members of the Joint Steering Committee representing AANDC Saskatchewan region, with the CEU, and with YTC. The project also lacked a performance measurement framework and reporting was activity-based. Project outcomes were not tracked or reported systematically. Despite this, the evaluators were able to obtain sufficient evidence to triangulate the evaluation findings.

2.4 Roles, Responsibilities and Quality Assurance

The evaluation was performed by EPMRB with the assistance of Stiles Associates Inc. Three members of the Joint Steering Committee representing AANDC Saskatchewan region, Yorkton Tribal Council and the CEU were consulted in the preparation of the evaluation methodology. They also validated the preliminary findings of the evaluation and reviewed the draft evaluation report.

2.5 Data Analysis

When presenting qualitative data in this report, certain terms are used to indicate the proportion of respondents or case studies to which the finding refers. Other terms indicate the frequency with which respondents expressed a particular view. These terms are roughly equivalent to the following percentages:

Proportional Term	Frequency Term	Percentage
All	Always	100
Almost all	Almost always	80-99
Many	Often, usually	50-79
Some	Sometimes	20-49
Few	Seldom	10-19
Almost none	Almost never	1-9
None	Never	0

Findings may not apply to certain respondents or case studies because no response was provided or no response could be inferred from other comments; or because a different response was provided. Where possible, alternative views as well as the views of the majority are presented.

The evaluation looked for evidence that the Comprehensive Community-based Planning pilot project is consistent with federal roles and responsibilities, responds to the needs and priorities of Saskatchewan First Nations communities, and does not duplicate or overlap with other departments or jurisdictions.

Findings from the evaluation show that comprehensive community-based planning is consistent with federal roles and responsibilities as it aligns with AANDC and government priorities. All stakeholders interviewed, especially First Nations involved in the pilot project, believe there is a need for comprehensive planning that facilitates community engagement and ownership. Some of the literature and some respondents see comprehensive planning contributing to improved governance and self-reliance in First Nations communities. All lines of evidence point to the need for AANDC to move from "silo," program-based planning to a model that is both comprehensive (holistically examining community needs and priorities) and community-based (allowing for broad-based engagement of community members in the planning process). The evaluation found no evidence of duplication or overlap between the pilot project and planning efforts by AANDC or other jurisdictions.

3.1 Consistency with Government Priorities

The Comprehensive Community-based Planning pilot project remains highly consistent with AANDC's mandate to "develop, healthier more sustainable communities". The pilot project delivered on a commitment made in AANDC's Sustainable Development Strategy 2007-2010 to support First Nations community planning. By enabling communities to articulate and document their long-term vision, needs and priorities, comprehensive community-based planning supports the requirements contained in the federal government's Policy on Transfer Payments that AANDC take a more citizen-focused, risk-based approach to how it manages transfer payments to Aboriginal communities.

For the period 2011-2014, AANDC has three departmental planning priorities: 1) transforming for improved results; 2) improving partnerships and relationships; and 3) managing resources effectively. To support the second priority, the Department is developing a new Community Development Framework that is more community-driven and more responsive to community needs than before. Community-level strategic planning has been identified as one of the potential tools supporting such a shift.

Comprehensive community-based planning supports the federal government's commitment to reduce the reporting burden facing Aboriginal communities by modelling a more comprehensive approach. The Auditor General of Canada identified this need in 2002 as did the Blue Ribbon Panel on Grants and Contributions Programs in 2006. Comprehensive community-based plans could potentially be used to assist AANDC in streamlining programming, thus, supporting the Department's priority of "managing resources effectively."

3.2 Need

Numerous sources, including the Royal Commission on Aboriginal Peoples, have documented the persistent challenges confronting First Nations communities such as high rates of unemployment, poverty, health problems and rapid population growth. There is growing realization that the multiplicity of nationally designed and individually administered programs dealing with specific issues is inefficient and often ineffective. A fragmented approach to planning and programming fails to examine the interconnections among the challenges facing First Nations communities and puts pressure on the limited human resources capacity available in small First Nations, some of which may have fewer than 1,000 members.

All lines of evidence point to the need for AANDC to move from "silo," program-based planning to comprehensive planning across all areas of importance to First Nations using processes that allow community members to take an active role in articulating community needs and priorities. Canadian and international literature from community planning (CEU 2010, Cook 2008, Copet 2003, Harivel and Anderson 2008, Public Works and Government Services Canada 2004, Wesley-Esquimaux and Calliou 2010, Wolfe 1989), environmental health (Lee 2002), Aboriginal economic development (Cornell and Kalt 1998), psychology (Thurman et al. 2003), and international development (World Bank 2009, Mathie and Cunningham 2002) support comprehensive, community-driven development approaches. In the late 1960s, 1970s and early 1980s, AANDC supported a number of comprehensive community planning initiatives. However, the success of those initiatives was limited due to a range of problems, including low levels of funding, incomplete implementation strategies, bureaucracy that hindered program effectiveness and the use of external consultants with limited understanding of Aboriginal cultures. (Copet 2003, Wolfe 1989)

All of those interviewed for this evaluation agreed there is a need for a planning model that is comprehensive and that facilitates community engagement and ownership. The case studies provide further evidence, as do AANDC evaluations of similar comprehensive planning initiatives in the Atlantic region and British Columbia. Some of the literature (Copet 2003, Wolfe 1989, CEU 2010) and some respondents from First Nations, tribal councils and Government spoke of the contribution comprehensive community-based planning can make to improved community governance and self-reliance.

Furthermore, several respondents and some of the literature regard comprehensive community‑based planning as culturally appropriate because it supports the type of consensus approach to decision making that is practiced in many Aboriginal communities.

3.3 Duplication

The evaluation found no evidence of duplication or overlap between the pilot project and planning efforts by AANDC Saskatchewan region or other jurisdictions. First Nations, AANDC and other government departments produce a vast array of plans, including capital plans, infrastructure plans, housing plans, education plans and health plans. But none of these take both a holistic approach to examining community needs and engage community members in articulating their priorities.

Flowing from the Policy on Transfer Payments, AANDC now conducts annual General Assessments that identify the strengths and emerging risks of Aboriginal communities receiving AANDC funding. [Note 4] While these assessments do try to take a comprehensive view, they are an AANDC requirement and do not flow from a community-driven process to articulate the needs, vision and priorities of community members.

The evaluation looked for evidence that the project objectives and outcomes were clear to all stakeholders, the project design was appropriate, there were sufficient and appropriate resources and support to implement the project and there was adequate flexibility to allow the project to adapt to local contexts.

The evaluation found that while most pilot communities understood the project objectives, some had a different understanding that led to uncertainty about AANDC Saskatchewan region's intentions. The project design by the CEU was appropriate and provided an effective, interactive planning model that, with the assistance of a paid Plan Champion, facilitated broad-based community engagement. However, some communities expressed concern that they had little say in the project design, that planning was limited to the boundaries of the reserve, and that it focused too much on physical planning and not enough on broader social and economic issues. While there was good representation of women and men in the planning process, the project failed to incorporate attention to gender equity issues, such as the need to explore the differing needs of women and men in planning and gender impacts in implementation.

Evidence showed that the CEU fulfilled the terms of its contracts. It smoothly rolled out the seven-stage model by establishing strong relationships with communities. All stakeholders agreed that the Unit was flexible in adapting the project to the local context in different communities. There appear to have been sufficient resources for planning, but many communities expressed concern about the lack of support from AANDC Saskatchewan region to implement the community projects and priorities identified in the plans. Other delivery challenges included a lack of engagement by AANDC Saskatchewan staff, the distance between Saskatchewan pilot communities and CEU's offices in Halifax and insufficient time for the Phase 3 communities that joined the project in 2008-2009 to complete the planning and implementation process.

4.1 Design

Evidence from the document review, case studies and interviews showed that most of those involved with the pilot project held a shared view of project objectives, which was to involve communities in comprehensive, interactive planning that engages community members and sets a vision for the future. However, this understanding was not shared by all pilot communities. The lack of a logic model meant outcomes were not clearly articulated and this, along with the pilot nature of the project, created some uncertainty about AANDC Saskatchewan region's intentions. Some First Nations were unclear about how the community plans would be used by AANDC Saskatchewan region and were concerned that such planning would become a departmental requirement for future funding. However, others felt that community plans should be used by AANDC Saskatchewan region to determine funding requirements.

Project stakeholders found that the seven-stage planning model used by the CEU was effective in facilitating community engagement and producing high-quality community plans that take a comprehensive view of the community, including health and wellness, education, governance and communication, infrastructure and housing, economics, justice and safety, and the environment. Community members described the process as "user-friendly", "hands-on" and "interactive".

The case studies, interviews and document review showed that hiring a community member as a paid Plan Champion was key to fostering broad-based engagement in the planning process. In each of the five case studies, the evaluators found evidence that a wide cross-section of community members took part in planning, including women, men, elders, youth, community leadership, and band members who would not usually be involved in such an initiative. The Plan Champion organized community events, produced newsletters and other information materials, and, in some cases, paid individual visits to each home on the reserve to explain the planning process and get band members involved.

One gap in the project was the lack of attention to gender equality issues as required under AANDC's Gender-based Analysis Policy. AANDC Saskatchewan did not include gender equality considerations in its request for proposals. While there was good representation of women and men on the Planning Work Group and in the planning process, neither CEU's planning model nor the actual planning recognized the necessity to explore the differing needs of women and men and the differential gender impacts of the plans developed. 

For Phase 2 and Phase 3 communities, Plan Champions from earlier phases acted as mentors to the Plan Champions in the newer communities. The case studies and interviews showed this peer-to-peer learning process worked well in providing ideas and support in cases where the mentors had been effective Plan Champions in their own communities.

While the pilot project was set up to explore a community-based planning model, the evaluators found the project was not entirely community-driven. With the assistance of a Technical Advisory Group made up of representatives of First Nations and tribal councils, AANDC Saskatchewan region selected the CEU to carry out the initial phase of the pilot project. According to an AANDC Saskatchewan official, the CEU was selected because it had a documented model for comprehensive community-based planning and a track record in First Nation communities in the Atlantic region. While the Technical Advisory Group helped establish the criteria for selecting pilot communities to participate, AANDC Saskatchewan region selected the communities and invited them to participate.

Community members in some pilot communities said they would have liked to have had a greater say at the outset in how the project was designed. They were concerned that planning was limited to the physical boundaries of the reserve. While the CEU emphasized that communities had the opportunity to determine the extent of their plans, some community members said they would have liked to "think outside the box" to include traditional lands and new land acquisitions, consider members living off reserve and engage provincial and municipal governments in the planning process.

The CEU model tends to emphasize physical planning solutions – buildings and infrastructure as a tangible starting point for the process of community change. However, some community members and other stakeholders would have liked to have seen more emphasis on health and social issues, and the economic issues underpinning community challenges.

The communities invited to participate were diverse in size and geographically distributed across the province, but all had comparatively strong financial and human resource capacity (with the exception of one community). A more representative or stratified sample of Saskatchewan First Nation communities would have provided more meaningful information for learning, particularly in relation to the model's reliability for broader application.

According to its terms of reference, the Joint Steering Committee was to manage and oversee the pilot project, but in practice it acted as an information, knowledge exchange, and training forum rather than as a governance body. Most participants found it useful for sharing and learning from the experiences of others. It also served as a forum to brainstorm on future support to comprehensive community-based planning such as the outlines of a proposed community planning network. In Phase 1, the project organized Federal Days during Joint Steering Committee meetings, which raised awareness about comprehensive planning beyond AANDC Saskatchewan representatives and allowed communities to make contact and search out funding and partnerships with other departments to collaborate on specific projects. Representatives of Phase 1 pilot communities saw the Federal Days as the most useful aspect of the Steering Committee. But community representatives from later phases were disappointed that they did not have the opportunity to participate in the Federal Days because the event was eventually wound down due to lack of engagement from federal departments.

4.2 Delivery

Evidence from the case studies, interviews and project documentation show the CEU fulfilled the terms of its contract. CEU staff rolled out the seven-stage model smoothly. They established strong relationships with community members that facilitated community engagement. Case studies and interviews showed that the CEU was able to adapt to the local culture and differing conditions in pilot communities. Community members and leadership spoke highly of the approach taken by the Unit staff, describing them as "open", "approachable" and "respectful."

However, many pilot communities would have liked to have had the project delivered by Aboriginal consultants or consultants with a stronger foundation in local culture. Some First Nations respondents said the time and physical distance between Saskatchewan and the CEU offices in Halifax diminished opportunities for on-going interaction and created logistical challenges. There were times when community stakeholders had difficulty contacting Unit staff and the project faced challenges rescheduling workshops that had to be postponed due to unexpected events in communities, such as funerals.

There was broad agreement that the project provided sufficient human and financial resources to develop the plans. However, many community stakeholders were critical of the lack of AANDC Saskatchewan region funding and support for implementation of the community projects and priorities identified in the plans. Phase 3 communities felt they had insufficient time to complete the planning process and implementation process, and two of thesecommunities have yet to publish their plans. At the outset of the project, Phase 1 communities were to receive three years of support. However,these communities successfully lobbied AANDC for further support for plan implementation and by the end of the project, AANDC Saskatchewan region provided up to six years of support for Phase 1 Plan Champions while Phase 3 communities received two years or less.

While there was strong engagement of AANDC Saskatchewan staff at the beginning of the project, it declined over time and, for the most part, the Department's participation was limited to attending Steering Committee meetings. Staff turnover at AANDC Saskatchewan region affected momentum and engagement with three different managers being responsible for the pilot over its six years of operation. While many AANDC staff were involved in Phase 1 of the pilot project, with one exception, current funding services officers, AANDC Saskatchewan region's front-line staff, had little awareness of the planning process. Given that funding for the project was not A-based but pulled together from the AANDC Saskatchewan region's Professional and Institutional Development Program, the Capital Program and Community Economic Development Program and from Health Canada [Note 5], there was uncertainty about funding from year to year. In at least one case, this led to the loss of an experienced Plan Champion who left to take on a more secure position.

The evaluation examined the expected and unexpected results the project achieved and whether the benefits would be sustained beyond the end of the pilot project.

Overall, the evaluation found the project achieved its expected outcomes. It raised awareness of the value of comprehensive community-based planning in pilot communities, with tribal councils and AANDC Saskatchewan region, increased the capabilities of pilot communities to engage in the development and use of community plans, and led to improved planning.

The project built cohesion and trust as community members came together to document their history, articulate their values and develop a joint vision for the future. The majority of communities are using their plans and almost all have implemented at least one of the initiatives identified in the plan. Some have developed new community infrastructure such as gardens or walkways, initiated new partnerships or made changes to community governance. The planning process also created greater expectations that community political leaders will inform and consult with community members. Plan Champions built new skills in community facilitation and engagement and increased their self-confidence, which helped most to move on to other jobs.

However, there is little evidence that the comprehensive community-based plans are being used within AANDC Saskatchewan region. There appeared to be little awareness of the plans among current front-line funding services officers and there is no evidence that the plans have been used to inform AANDC Saskatchewan region funding priorities for the pilot communities.

The pilot project produced a number of unexpected results, including economic benefits and the revival of cultural traditions.

The evaluation found that some of the benefits from the planning process will be sustained. In some pilot communities, the plans have provided stability of vision through several elections for Chief and Council. However, progress on initiatives identified in the plans, which often fall outside or across responsibilities in the band administration, will likely slow down without a Plan Champion within the community to move them forward. Furthermore, the technical expertise for the pilot project came from outside the province and the project has not been successful to institutionalize the capacity for comprehensive community-based planning in Saskatchewan.

5.1 Achievement of Outcomes

The evaluation found that the pilot project achieved significant results, including all of its expected outcomes. Evidence from the case studies, interviews and documentation showed that broad-based community engagement in pilot communities increased awareness of the importance of planning among community members, staff and elected leaders. Through the planning process, communities were able to come together and document their history, articulate their values and develop a common vision for the first time.

In many communities, the planning process increased community cohesion and built trust. For example, in one community, discussing strengths and weaknesses, identifying the root causes of community challenges, and developing the community vision helped it to deal openly with long-standing divisions and work toward unity.

Evidence shows that the project also raised awareness of the value of comprehensive community-based planning among the staff of AANDC Saskatchewan region, Health Canada and tribal councils who participated in the process. It also created interest among other First Nations in the province, some of which are eager to undertake their own planning process. Many stakeholders, inside and outside of AANDC, praised AANDC Saskatchewan region for having the foresight to support a pilot project that incorporates internationally established good practice in community development.

The project increased the capabilities of community members to engage in and use comprehensive community-based plans. Community members told the evaluators that involvement in both the planning process and implementation created a sense of pride and belonging that helped band members feel better about themselves and their communities. Communities are proud of their plans, which are professionally printed, easy to read and illustrated with maps and photos. While each plan is unique, all follow a standard template developed by the CEU that includes three parts: context on where the community is now, vision about where the community is going, and action areas for shaping the future.

Three communities failed either to publish their plans or to move forward with implementation. In two communities, this inaction was largely because the political leadership did not agree with the proposed direction outlined in the plans. The third community was slow to get started and felt it was rushed through the planning process. Despite this, community members told evaluators that bringing the community together through the planning process still provided benefits in connecting the community and demonstrating the role community members could play in planning.

Plan Champions built new skills in community facilitation and engagement and increased their self-confidence. Many of the Plan Champions were women and most were able to move on to other jobs following their work with the project. One told evaluators that she is contemplating running for Band Council, something she would have never considered before her work on the community plan.

The evaluation found that almost all of the pilot communities have used their plans to some extent and many communities have made progress implementing them. In five of the eleven communities, the CEU (in four cases with the help of Dalhousie architecture students), worked with community members to design and build a new structure in the community. These structures included a pow-wow arbour, a green-shed, an environmental pavilion, an outdoor classroom and a community market. The projects provided communities experience in using local resources, training people, increasing ownership, being creative about funding support and partnerships, such as donated or recycled materials and volunteer labour to implement a project.

Through their own efforts some communities have gone much further. For example, one community developed a walking path from the village to the school that is lit with streetlights so that children do not have to walk to school along a busy highway. Furthermore, it organized a tree planting day, and obtained funding to build a veteran's memorial and buy new equipment for its senior's centre.

According to the interviews and case studies, comprehensive community-based planning has allowed some communities to challenge the current federal investment model whereby First Nations respond to centrally designed government programs. Some communities are using their plans to communicate their own priorities and tell potential funders where investments need to be made in their communities. In this way, the pilot project helped some communities to develop new partnerships and leverage funding. For example, in one community, the band used the community plan to support a successful proposal to the Saskatchewan Government for funding to construct a $1.6 million Centre of Excellence for Business Development that will provide business support and skills training. Another community developed partnerships with an international non-profit organization and the Saskatchewan Department of Agriculture to set up an organic community garden.

The planning process allowed communities to openly discuss governance issues such as lack of transparency and nepotism, and several communities have made changes to governance as a result. Two communities have developed and ratified new election acts, and one of those has also developed a new housing policy and financial management act. In another community, the band administration has tried to improve its communication with members through newsletters and Internet and is in the process of developing a community radio station. The same community is also working to update its bylaws.

In many communities, the evaluators were told that the planning process has increased expectations that the leadership will inform and consult with community members. One tribal council provided support for the political leadership of a pilot community to attend nation building training to assist the Chief and Council in learning how to respond to these expectations.

While most pilot communities are using their plans, there has been little use of the plans by AANDC Saskatchewan region. The one exception was in the development of the General Assessments that are now required for all communities receiving AANDC funding. When the evaluators spoke to current funding services officers who deal directly with the pilot project communities, they learned that the majority had never seen or used the community plans. Furthermore, there was no evidence that AANDC Saskatchewan region is using the community plans to inform departmental funding priorities.

The approach in AANDC Saskatchewan region contrasts with that in AANDC's British Columbia (BC) region, which provides support both to communities in developing comprehensive community plans and to AANDC staff in responding to those plans. In BC region's Strategic Planning and Communications Directorate, one and a half staff positions are devoted to providing guidance, support and advice related to comprehensive community planning. Part of the work of the staff members includes pulling together teams from different AANDC BC programs to examine how to support the priorities identified in community plans. The Directorate also conducts regular training sessions on comprehensive community planning for funding services officers.

A 2010 evaluation of Comprehensive Community Planning in the BC region (KTA Inc. and Naut'sa mawt Resources Group 2010) showed that it had accomplished three goals – increased awareness, increased capacity, and strengthened relationships between First Nations and AANDC BC. The Saskatchewan pilot project also increased awareness and capacity. However, evaluators only found evidence of a strengthened relationship between AANDC Saskatchewan region and a First Nation in one pilot community. In that case, the funding services officer was directly involved in the planning process and the community leadership said that the process had facilitated a more open and positive relationship with the Department.

Tribal councils are tasked with providing advisory and support services to member bands and council representatives. They were included in the pilot project as one means of building awareness and capacity for comprehensive community-based planning within the province. While some tribal council representatives were actively engaged in the planning process, others felt there was insufficient funding to allow them to participate in more than the occasional Steering Committee meeting. One tribal council representative is using elements of the comprehensive community-based planning process in his work with communities. However, given the high demand for planning specialists in the province, others were sceptical about the ability of tribal councils to recruit planning specialists with the technical capacity to support comprehensive community planning efforts.

5.2 Unexpected Results

The pilot project produced a number of unexpected results – both positive and negative. By facilitating community participation, the process allowed community members who would not normally participate in planning to get involved. Community members got excited about potential ideas and, in some cases, moved forward on their own to put those in place. For example, seniors in one community came together to set up a restaurant and gathering place in an unused space in the community mall.

The planning process produced a number of economic and cultural benefits. Through the community garden project mentioned above, one community was able to get eight people off social assistance for two years and provide them with training in organic gardening. Another community was inspired to revitalize its annual pow-wow after a 20-year lapse. That pow-wow is now an established annual event on the pow-wow trail in Western Canada.

The community plans have won two awards of excellence from the Canadian Institute of Planners, including an award for rural and small town planning in 2008 and another for social planning in 2011. The CEU has showcased the plans internationally and an educator from one pilot project community shared his community's plan with indigenous education officials in Australia.

An unexpected negative result identified by the case studies and interviews was the creation of expectations for quick action on the projects identified in the plans – expectations that were not fulfilled. This outcome was most prevalent in communities that have yet to follow through with plan implementation. While AANDC Saskatchewan region was clear at the outset that the pilot project would not include funding for community projects, the very act of funding the planning process created expectations that the Department would respond to the priorities identified in the plans. Furthermore, some community leaders questioned the feasibility of some of the projects identified in the plans, given local circumstances and the limited availability of funding. Despite the creation of expectations, some stakeholders in this project regard it as an inevitable, and not necessarily negative, outcome of a process that invites community members to define their vision of a better community.

5.3 Sustainability

Evidence from the case studies, interviews and document review shows that some of the benefits of the pilot project will be sustained. Most communities are still using their plan and some have integrated it into the work of their band administration. One community prominently displays the value statements from their plan in the band office, community school and adult education centre. For at least four of the pilot communities, the community plan has provided a stable vision that has continued to be supported through several elections.

The evaluators found that three of the communities that made the most progress in implementing their plans were among those in Phase 1 of the pilot project. However, it is difficult to say whether this is due to the sustained support these communities received over six years or whether their uptake of the plans grew out of their strong human and financial resource capacity that was there before the pilot project.

The broader sustainability of the planning process is, however, in some doubt because AANDC Saskatchewan region has yet to follow through in using the plans to inform programming priorities. The CEU process envisions that First Nations will, over time, begin to change their governance structures to align with the priorities identified in their community plans. However, in the meantime, band administrations remain largely partitioned in health, education and other compartments to align with federal government funding programs. Some stakeholders emphasized how profoundly different the comprehensive, community-based model of planning is from typical strategic planning exercises that focus only on a specific sector. They also underlined that moving toward comprehensive, community-based planning is a major change in approach and thinking for federal departments used to the narrower strategic planning model.

The evidence suggests that there would be a loss of momentum on plan priorities in communities once the funding for the Plan Champion ends. Since the action areas identified in the plans often fall outside or across responsibilities in the band administration, community stakeholders believe a Plan Champion is vital to drafting the funding applications or to organizing the community members needed to move those priorities forward.

The design-build projects that formed part of the CEU model to kick-start plan implementation provided positive learning experiences that created pride and a sense of accomplishment. However, in some communities, the structures have yet to see much use by community members and in one community, the structure had to be dismantled because it was vandalized. The design‑build projects appear to have been selected quickly by communities without comprehensive needs assessments. Overall, projects such as the community walkways that the communities themselves initiated and developed appear to have greater sustained use.

CEU received funding as part of the final phase of the project to work toward establishing a First Nations Planning Network among the participating communities and tribal councils. However, there was no indication that the network would continue once project funding ended since neither the communities nor the tribal councils possess the technical expertise or resources to sustain such a network.

Over the course of the project, the CEU submitted two proposals to AANDC Saskatchewan region for funding to establish a diploma program in First Nations community planning, first with the First Nations University in 2006 and then with the First Nations University and the University of Saskatchewan in 2010. Such a program was to provide on-going access to local planning expertise in Saskatchewan. While AANDC Saskatchewan region set aside funding in 2010-2011 to support such an initiative, it has yet to become a reality.

The evaluation sought to determine whether the most appropriate and efficient means were used to develop the community plans and whether alternative models could improve the process or reduce the cost.

Overall, the planning process was funded at an average cost of about $450,000 per community. The fees paid to the CEU appear reasonable for the services delivered when compared to rates charged by planning consultants. However, the travel between the CEU office in Halifax and Saskatchewan added to the project costs, as did expenses associated with Joint Steering Committee meetings.

Building technical capacity to support First Nations comprehensive community-based planning in Saskatchewan would both increase efficiency and reduce costs over the long-term. Alternative delivery models and approaches for comprehensive community-based planning and working with geographic clusters of communities could result in cost savings.

6.1 Economy

AANDC Saskatchewan region and Health Canada provided significant resources for the pilot project – on average about $450,000 per community when the $5 million cost is divided by the 11 participating communities. With this funding, the project was able to hire a Plan Champion in each community, take the time and effort needed to facilitate broad-based community involvement and in many communities support plan implementation.

The funding for the pilot project was roughly divided between support to First Nations (50.7 percent to communities and tribal councils) and the contract with the CEU (44.7 percent), which is in line with CEU's view that the level of effort and activity was equally distributed between CEU and First Nations. However, some First Nations expressed concern about the proportion of funding that went to the CEU.

An examination of the fees charged by the Unit shows that they appear reasonable for the services delivered as they were lower than rates normally charged by professional planning consultants. However, the distance between Halifax and Saskatchewan added to the cost of the project. While the total travel costs are not broken down in CEU contracts, budget documents show that airfare alone comprised nearly $150,000, or six percent of Dalhousie's costs.

The Joint Steering Committee meetings were also costly, bringing together between 15 and 50 participants from around the province for one- or two-day meetings several times a year. The meetings appear to have cost about $260,000 and comprised about five percent of the project budget. The evaluators found the benefits of these meetings difficult to measure. However, most participants found it useful for sharing and learning from the experiences of others. It also served as a forum to brainstorm on future support to comprehensive community-based planning such as the outlines of a proposed community planning network. In Phase 1, the project organized Federal Days during Joint Steering Committee meetings that raised awareness about comprehensive planning beyond AANDC Saskatchewan representatives, and allowed communities to make contact and search out funding and partnerships with other departments to collaborate on specific projects. Representatives of Phase 1 pilot communities saw the Federal Days as the most useful aspect of the Steering Committee.

One of the lessons from international development is that there are no shortcuts to effective community engagement and it can, therefore, be costly to undertake. (Mansuri and Rao 2004) However, when done well, the benefits in program sustainability outweigh the costs.

6.2 Efficiency

Representatives of communities, tribal councils and AANDC all agreed that there is a need to build technical capacity for First Nations comprehensive community-based planning within Saskatchewan. Access to local participatory planning expertise would reduce the travel costs and increase ease of access. Developing that technical capacity will require a long-term funding commitment. Many stakeholders emphasized the need for such a commitment to allow comprehensive community planning to move beyond pilots to being accessible to all First Nations in the province.

Some of those interviewed also suggested that AANDC Saskatchewan needs to explore alternative models or approaches beyond the CEU model for delivery of comprehensive community-based planning. Some stakeholders suggested that the proposed diploma and planning network initiatives may contribute to mitigating the cost of producing community plans. Another idea in this respect is to examine AANDC BC's approach, whereby, the Department provides funding, tools and support while communities wishing to undertake comprehensive community planning identify the model and technical expertise required to fit their needs.

When questioned about the pilot project's delivery over three phases, most community stakeholders thought it would have been preferable to proceed with all 11 communities from the start of the project. This would have given all communities sufficient time to complete the planning and implementation process. Some suggested it would have been more efficient to work with clusters of communities, for example, several communities within a tribal council rather than separate communities scattered around the province. Such an approach would allow for greater information sharing and support between communities. Further, community members suggested that involving other levels of government in the planning process could assist in developing partnerships.

The evaluation looked for evidence that the pilot project had mechanisms in place to capture lessons and facilitate learning and examined lessons that could contribute to improved community planning.

The evaluation found limited evidence that mechanisms were in place to capture lessons and use them to facilitate reflection and on-going learning at the community level. The CEU worked with pilot communities to develop indicators to track progress in meeting community goals. However, neither it nor AANDC Saskatchewan region demonstrated to communities how monitoring data could be collected and used to inform on-going planning and program improvement. The evaluation identified a number of lessons and success factors that are outlined below.

7.1 Learning

The evaluation found limited evidence that mechanisms were in place to capture lessons and use them to facilitate on-going learning at the community level. The lack of a project logic model and performance measurement framework meant that the CEU did not develop indicators against which it could collect data, track progress and make adjustments to the planning process as the project progressed. AANDC Saskatchewan region only required activity-based reporting and this meant that data such as the level of community participation and the changes resulting from the planning process were not tracked systematically, reported or examined to see what could be learned.

As part of its work with pilot communities, the CEU did discuss the need for on-going monitoring of plan implementation to track progress in meeting community goals. It also worked with communities to develop indicators of success that were to be tracked annually. However, these monitoring sections often lacked causal linkages to the actual projects that communities undertook and there is no indication that communities were using the indicators to monitor the implementation of their plans. Both AANDC Saskatchewan and CEU missed an opportunity to demonstrate to communities how monitoring could be used for on-going learning and improvement.

Facilitating learning requires more than the collection of data; it requires a reflective practice that sets time aside to reflect on progress, identify lessons and apply them, usually on an annual basis. Although CEU uses reflective practice in its own assessment of the project (CEU 2010), such practice does not appear to have been integrated into its work during the project at the community level.

Recent literature on comprehensive community initiatives that deal with complex and interconnected problems, such as those facing First Nations communities, point to the contributions real-time monitoring and evaluation can bring to those processes. (Gardner 2011) According to the literature, there is a need to build evaluation and learning into such processes as an enabler of innovation, continuous service improvement and community learning, rather than only as a means to meet narrow accountability requirements. As comprehensive community based planning progresses, it will be important to work with communities to develop meaningful and culturally appropriate monitoring and evaluation tools that allow communities to assess their own progress and make improvements.

7.2 Lessons

The case studies, interviews, and project documentation provided a number of lessons related to comprehensive community-based planning in Saskatchewan that are also supported by the literature review.

• Planning must be community-based

It is only by engaging community members in meaningful dialogue and decision making that planning can be successful in moving a community forward. Engagement creates awareness, builds capacity, fosters cohesion, and raises expectations.

• Planning must be comprehensive

Planning must view the community as a whole, taking into consideration the interconnections between health, social, economic, land use, infrastructure and environmental issues. This implies moving from a program-based approach to a community development approach.

• Both AANDC and communities must make changes to develop a new relationship

Both AANDC and First Nations communities must make changes if comprehensive community‑based planning is to achieve its potential. A plan becomes a tool for crafting a new relationship when the Department responds to community priorities. Such a shift from directive to responsive programming requires a fundamental change in the way the Department conducts its business.

• Comprehensive community-based planning takes time and requires long-term support

This type of planning is part of a long-term process of community development. It requires sustained support to build awareness, capacity and community engagement. Not all communities will proceed at the same pace, as the pilot initiative has shown.

• Comprehensive community-based planning is costly, but potentially cost-effective

While this type of planning is costly to do well, it is potentially cost‑effective since it can help direct spending where it will be most effective in responding to community needs.

7.3 Success Factors

Over the course of their research, the evaluators identified a number of factors that contributed to the results achieved by the comprehensive community-based planning project in Saskatchewan.

• A paid Plan Champion

Hiring a community member to facilitate the plan development is key to providing momentum, engaging community members and moving the process forward.

• Planning expertise combined with good process and skilled facilitation

Supporting the development of comprehensive community-based plans requires a strong grounding in community planning along with a good process and excellent facilitation skills to engage community members, and allow them to openly deal with difficult issues and envision a way forward.

• Starting from strengths

An asset-based approach that starts from community strengths rather than from problems helps people to open up and reduces negativity – all of which puts the community in a better position to move forward.

• Engaging elders and youth

Planning allows elders an opportunity to contribute and helps ground the process in the culture. Engaging youth is crucial, but finding ways to do so requires innovation.

• Ensuring political leadership is on-side

Chief and Council have to understand from the outset the implications of comprehensive community-based planning and be involved throughout, without controlling the process.

• Providing support for leaders

If comprehensive community-based planning is to be successful, there may be a need to assist community political leaders in understanding the changes required in community governance in order to move forward with the priorities outlined in the plans.

• Building relationships and partnerships

Building new relationships and partnerships beyond AANDC and the federal government with the private sector, non-governmental organizations, provincial and municipal governments is key to meeting community needs.

• Mentoring

Peer-to-peer learning can be an effective and potentially cost-effective way to share expertise between communities.

8.1 Conclusions

The evaluation set out to provide evidence-based conclusions regarding the relevance and performance of the Comprehensive Community-based Planning pilot project in Saskatchewan between 2005-2006 and 2010-2011. Given that most of the communities selected to participate in the pilot project had good financial and human resource capacity, the findings on the development of this planning model may be limited to such communities.

The evaluation findings support the following conclusions.

Relevance

Findings from the evaluation show that comprehensive community-based planning is consistent with federal roles and responsibilities as it aligns with AANDC and government priorities. All stakeholders interviewed, especially First Nations involved in the pilot project, believe there is a need for comprehensive planning that facilitates community engagement and ownership. Some of the literature and some respondents see comprehensive planning contributing to improved governance and self-reliance in First Nations communities. All lines of evidence point to the need for AANDC to move from "silo," program-based planning to a model that is both comprehensive (holistically examining community needs and priorities) and community-based (allowing for broad-based engagement of community members in the planning process). The evaluation found no evidence of duplication or overlap between the pilot project and planning efforts by AANDC other jurisdictions.

Performance

Design and Delivery
The evaluation found that the project design by the CEU was appropriate and provided an effective, interactive planning model that, along with the assistance of a paid Plan Champion, facilitated broad-based community engagement. However, some communities expressed concern that they did not have a say in the project design, and that planning was limited to the physical boundaries of the reserve. Further, AANDC Saskatchewan region failed to ensure that the project incorporated attention to gender equity issues, as required by AANDC's Gender-based Analysis Policy.

The CEU successfully delivered the pilot project and adapted it to the local context in different communities. There appears to have been sufficient resources for planning and implementation, but many communities were concerned about the lack of financial support for the projects and priorities identified in their plans.

Effectiveness
The evaluation found that the project achieved its expected outcomes. It raised awareness of the value of comprehensive community-based planning in pilot communities, with tribal councils and AANDC Saskatchewan region, increased the capabilities of pilot communities to engage in the development and use of community plans, and led to improved planning.

The project built cohesion and trust as community members came together to document their history, articulate their values and develop a joint vision for the future. The majority of communities are using their plans and almost all have implemented at least one of their planned initiatives. Some have developed new community infrastructure such as gardens or walkways, begun new partnerships or made changes to community governance. Plan Champions built new skills in community facilitation and engagement and increased their self-confidence. However, there is little evidence that AANDC Saskatchewan region is using the comprehensive community‑based plans.

The evaluation found that some of the benefits from the planning process will be sustained. In several pilot communities, the plans have provided stability of vision through several elections. While the building of structures through the design-build process was a positive experience for communities, some structures have seen limited use. The technical expertise for the pilot project came from outside the province, and the project has not been successful to institutionalize the capacity for comprehensive community-based planning in Saskatchewan.

Efficiency and Economy
Overall, the planning process was funded at an average cost of about $450,000 per community. The fees paid to the CEU appeared reasonable for the services delivered when compared to the rates charged by professional planning consultants. However, travel between CEU's offices in Halifax and Saskatchewan added to project costs, as did the expenses associated with Joint Steering Committee meetings.

Building technical capacity to support First Nations comprehensive community-based planning in Saskatchewan would increase efficiency and reduce costs over the long-term. Working with geographic clusters of communities might also deliver cost-efficiencies.

Learning and Lessons
The evaluation found limited evidence that mechanisms were in place to capture lessons and use them to facilitate reflection and on-going learning at the community level. The CEU worked with pilot communities to develop indicators to track progress in meeting community goals, it did not demonstrate effective monitoring practice. Research for the evaluation identified a number of general lessons and success factors related to the experience with the Saskatchewan pilot project.

8.2 Recommendations

It is recommended that AANDC, to support comprehensive community-based planning:

1. Consider ways to institutionalize long-term national support for such planning in First Nation communities.
2. Review its approach to:
   
   
   1. better integrate First Nations input on the design and approach used to develop community-based plans;
   2. involve other levels of government; and
      
      
   3. integrate gender-based analysis.
      
      
3. Consider means to increase capacity (inside and outside the Department) to support comprehensive community-based planning.
   
   
4. Commit to using comprehensive community-based planning to inform program and funding priorities.

Cities & Environment Unit. (2010). Learning from Saskatchewan: Charting a course for community planning in Canada. Halifax: Dalhousie University.

Cities and Environment Unit and Wagmatcook First Nation. (2003) First Nations community planning model, 2nd edition. Halifax: Dalhousie University.

Cook, J. (2008). Building on traditions of the past: The rise and resurgence of First Nations CCP. Plan Canada, 48, 13-16.

Copet, Wayne (2003). An approach to community planning in Aboriginal settlements. Canadian Journal of Native Studies, 25(2) 37-50.

Cornell, S. and Kalt, J. (1998) "Sovereignty and Nation-Building: The Development Challenge in Indian Country Today" American Indian Culture and Research Journal. Vol. 22 (3) 187-214.

Gardner, B, (2011) Comprehensive Community Initiatives: Promising Directions for "Wicked" Problems. Horizons. Policy Research Initiative. Retrieved March 2, 2011.

Harivel, C. and Anderson, C. (2008). First Nations comprehensive community planning – A good investment for Canada. Plan Canada, 48, 29-31.

KTA Inc. and Naut'sa mawt Resources Group. (2010). Indian and Northern Affairs Canada: Evaluation of aspects of comprehensive community planning in the BC region.

Lane, Marcus (2001). Indigenous land and community security: A radical planning agenda. Land Tenure Center, Working Paper No. 25. Madison: University of Wisconsin.

Lee, C. (2002). Environmental justice: Building a unified vision of health and the environment. Environmental Health Perspectives, 110 (April), 141-144.

Mansuri, G. and Rao, V. (2004) Community-based and -driven development: A critical review. World Bank Research Observer, 19(1) 1-39.

Mathie, Alison & Gord Cunningham. (2002), From Clients to Citizens: Asset-Based Community Development as a Strategy for Community-Driven Development. Occasional Paper, Coady International Institute, St. Francis Xavier University.

Public Works and Government Services Canada (2004). Sharing the story – comprehensive community planning – experiences in First Nations, Inuit and northern communities.

Thurman, P. et al. (2003) Community readiness: The journey to community healing. Journal of Psychoactive Drugs, 35(1), 27-31.

Wesley-Esquimaux, C and Calliou, B. (2010) Best practices in Aboriginal community development: A literature review and wise practices approach. The Banff Centre.

Wolfe, J. (1989) Approaches to planning in native Canadian communities: A review and commentary on settlement problems and the effectiveness of planning practice. Plan Canada, 29 (2), 63-79.

World Bank. (2009). IDA at work: Community-driven development: Delivering the results people need  . Retrieved March 27, 2011.

EVALUATION OF THE COMPREHENSIVE COMMUNITY-BASED PLANNING PILOT PROJECT IN SASKATCHEWAN

Methodology Report

February 20, 2011 version

1. Introduction

Indian and Northern Affairs Canada (INAC) is conducting an evaluation of the Comprehensive Community-Based Planning (CCBP) Pilot Project in Saskatchewan. This evaluation will provide evidence-based conclusions regarding the project's relevance and performance.

The purpose of the evaluation and use of the data it generates relates to a request from INAC Saskatchewan region for an independent evaluation to assess the results achieved by the pilot project, capture lessons and provide recommendations to inform future support for CCBP.

2. Pilot Project Description

Scope of Community Planning

Community planning is a process through which a community establishes a direction for the future and determines specific actions to realize that direction. [Note 6] This is accomplished in a way that is widely understood, accepted and appropriated by many members of the community. The resulting plan allows the community to make informed decisions about where to seek funds, how to spend limited resources, how to react to requests for action, how to protect the environment and how to provide new opportunities for residents; all factors that create a sustainable community.

The Need for Comprehensive Community-based Planning (CCBP)

Planning, as a tool and in the context of First Nation (FN) communities is about the future and relies on information from both the past and the present. This information is to help understand current circumstances and to collectively determine the need for change. FN communities face many challenges, including social, economic, environmental, and governance issues. One of the ways to meet these challenges is through CCBP.

CCBP is a community planning tool that mobilizes communities through group processes (e.g. workshops) to identify strengths and issues affecting the community. CCBP helps the community to establish a vision, long-term goals, priorities and an action plan. CCBP is accomplished in a way that is widely understood, accepted and appropriated by many members in the particular community. Ideally, the process is community-driven and locally owned and controlled. Implementation and accountability for results rest with the community.

CCBP is expected to encourage communities to examine the consequence of doing nothing or of taking deliberate steps in a particular direction, thus helping to avoid crises before they occur. CCBP is recognized as an instrument that has the potential to help FN communities set the stage to guide individual decisions, plan long-term, focus on how and where the community should grow while serving as a means of transparency and accountability. The success of community plans rely on changing attitudes and approaches at the local and regional levels with respect to FN community-management.

2.1 Project history

The CCBP pilot project in Saskatchewan represents a commitment to rethinking FN community development and responding to urgent issues and needs of FN communities. It is intended to overcome the following barriers:

• Lack of local examples illustrating that community-based planning makes a difference;
  
  
• Lack of local expertise to initiate, guide and sustain the planning effort;
  
  
• Lack of resources and technological capacity at the local level; and
  
  
• Lack of awareness of what planning is.

INAC, through its Sustainable Development Strategy, established a commitment to support the introduction of CCBP within FN communities. This comprehensive approach is intended to facilitate a shift from a program-based service delivery model where, in many instances, external priorities have been placed upon FN communities, to a process more consistent with FN communities' goals and aspirations as articulated through their membership.

In 2005-06, the Saskatchewan region contracted with the Cities and Environment Unit (CEU), a research-based community planning action group of the Faculty of Architecture and Planning, Dalhousie University, Halifax, Nova Scotia, to engage four FN communities in CCBP. The selected communities were the Flying Dust FN, Gordon FN, Kahkewistahaw FN and Shoal Lake Cree Nation, with support from their respectiveTribal Councils. These four communities constituted Phase 1 of a pilot project.

In October 2005, a CCBP Joint Steering Committee (JSC) was established as a decision-making body to oversee and coordinate CCBP activities within the selected pilot FN communities; it included representatives from the selected four pilot communities, Tribal Councils, Health Canada, INAC and the Federation of Saskatchewan Indian Nations (FSIN).

In 2007-08 Dalhousie won a tendered contract to continue and four more communities were added. These Phase 2 communities were Cowessess FN, Kinistin Saulteaux Nation, Muskoday FN and Pasqua FN.

In 2008-09, Phase 3 communities – Big River First Nation, Lac La Ronge Indian Band and Standing Buffalo Dakota Nation entered the pre-planning stage of the process.

Through consultations with FN and Tribal Councils, community development and planning was established as a long-term strategic priority for INAC Saskatchewan region. The pilot project was one of the key activities supporting this priority. At present there are 10 FN communities involved in CCBP. These are:

PHASE 1 COMMUNITIES	PHASE 2 COMMUNITIES	PHASE 3 COMMUNITIES
1. Flying Dust FN	1. Cowessess FN	1. Big River FN
2. George Gordon FN	2. Kinistin Saulteaux Nation	2. Lac La Ronge Indian Band
3. Shoal Lake Cree Nation	3. Muskoday FN*	3. Standing Buffalo Dakota Nation
4. Kahkewistahaw FN	4. Pasqua FN

*Muskoday FN, a Phase 2 community, withdrew from the pilot project in April 2010.

All Phase 1 and 2 communities have published their plans and are in various stages of implementing or kick-starting other projects identified in the plans. Examples of community kick-start and action projects range from walking trails to market gardens to a coffee shop, construction of a pow wow arbour and an environmental research pavilion.So far, one of the Phase 3 communities has published its community plan.

2.2 Project Profile

Principles for the selection of First Nations

In April 2005, the inaugural meeting of the Community Development and Planning Technical Advisory Group (TAG) was held in Saskatoon, Saskatchewan. TAG , which was established to provide recommendations relating to community planning and development within the region, reviewed opportunities for collaboration between INAC, FN and Tribal Councils.

TAG was then tasked with the responsibility of establishing principles to be considered in selecting FN communities to pilot comprehensive planning theory. TAG established the following six principles:

1. Pilot communities, through their council, must confirm a willingness and desire to advance CCBP.
   
   
2. Communities from the North and South should be represented.
   
   
3. A pilot community must possess demonstrated financial management capacity and be committed to the principles of accountability.
   
   
4. Technical expertise from Tribal Council advisory staff and planning consultants will be essential in the initial pilots to build comprehensive planning capacity.
   
   
5. Initial pilot communities must be affiliated with a Tribal Council to ensure linkages are developed and planning is supported in the long-term.
   
   
6. Pilot communities must have demonstrated a commitment to community engagement as part of their day-to-day operations.

Based on the foregoing, the four FN communities, previously mentioned, were recommended for the 2005-2006 fiscal year.

2.2.1 Approach

The CCBP approach centers on a philosophy that encourages and employs extensive community involvement throughout the development and the implementation of a community plan. This approach ensures that community members and band administration are fully aware of the planning process, have avenues for meaningful input and participate in implementation of projects. The three fundamental conditions of CCBP are that:

1. The Plan comes from the community.
   
   
2. The Plan is appropriated by the community.
   
   
3. The Plan inspires and motivates the community.

In addition, the CCBP approach is based on the following three major points:

1. A team-centred approach: This approach, leading to the creation of a local planning work group builds on individual strengths and expands local capacity while maintaining the project's collective goal and mandate.
   
   
2. Comprehensive view: In this fundamental approach, the CEU, Dalhousie University, assists communities in the development of holistic community plans that considers land use and physical economic development as well as incorporates social development and environment sustainability.
   
   
3. An action-oriented plan: This approach translates visions, ideas and priorities into tangible projects and includes concrete and specific actions needed to achieve them.

2.2.2. Planning Framework

The CCBP model, as developed by the CEU (Dalhousie University), establishes a conceptual framework and approach to change centred on a comprehensive and community-based philosophy. The model outlines a seven-stage process with distinct products at the end of each stage that together constitute the Community Plan. The approach is intended to develop relevant community plans with First Nations that help them drive immediate action based on a long-term vision.

1. Stage 1: Gathering Background Information - Collect basic facts and perceptions about the community so that aspects that stand out as high and low points can be identified.
   
   
2. Stage 2: Identifying Strengths & Issues - Concentrate on recording and understanding problems that need to be dealt with and opportunities that can be built on.
   
   
3. Stage 3: Researching Root Causes. - Explore strengths and issues to reveal the root causes and the consequences of no action.
   
   
4. Stage 4: Establishing a Vision - Establish a long-term, ambitious and appropriate direction for the community.
   
   
5. Stage 5: Building a Framework - Translate the vision, issues and values into a blueprint for concerted action in terms of policies, administration, priority action areas and physical improvements.
   
   
6. Stage 6: Developing an Implementation Strategy - Develop an implementation strategy and determine priority projects. Organize the necessary resources to realize the projects.
   
   
7. Stage 7: Monitoring the Plan & Projects - Evaluate the impact of projects individually, the effects of planning as a whole, and revise the Plan on an on-going basis.

2.3 Project Objectives and Expected Outcomes

The overall objective of CCBP pilot project is to make community-based planning (developing comprehensive local, long-term, and action-oriented plans) an on-going routine for FN.

The CCBP project is intended to engage community members in each stage of the planning process. It does so by increasing awareness among the various stakeholders including Tribal Councils, communities, and the federal government. It also seeks to increase the community's capacity to engage in planning. It is expected that this will position communities to take on challenges, explore opportunities and succeed in meeting their goals.

The First Nations Community Planning Model, published by the CEU and Wagmatcook First Nation (Nova Scotia), defines products, outcomes and expectations for planning that were followed in the CCBP project. However, since no formal logic model or performance measurement framework was developed for this pilot project, a logic model developed for the purposes of the evaluation appears in Appendix C.

2.4 Project Governance (Management), Key Stakeholders and Beneficiaries

Project Management and Key Stakeholders

Joint Steering Committee

• Manages and oversees the pilot project
  
  
• Comprised of representatives of First Nation communities (Plan Champions), Tribal Councils, INAC, Health Canada, and the Cities and Environment Unit CEU (Dalhousie University)
  
  
• Supports and facilitates the development of strategies, work plans, and resources
  
  
• Extends project work to other FN communities

INAC, Saskatchewan Region:

• Federal lead in providing resources (technical and other), to support the CCBP process
  
  
• Chair of the Joint Steering Committee (JSC), responsible for all meeting logistics and secretarial support including arranging meetings and maintaining and facilitating contact among members
  
  
• Facilitates and supports partnership development initiatives between academic, technical and other organizations (i.e. Dalhousie and First Nations University)
  
  
• Pursues linkages to inter/intra departmental federal government programming
  
  
• Coordinates initiatives supporting professional and capacity development
  
  
• Monitors CCBP Terms and Conditions
  
  
• Liaison between INAC Regional and Headquarters CCBP National Working Group
  
  
• Based on recommendations from the JSC, maintains responsibility for resource management and budget allocation decisions
  
  
• Advocates for change within the federal system to ensure CCBP is supported and appropriately resourced on a long-term basis
  
  
• Coordinates and administers funding arrangements with the pilot communities and affiliated Tribal Councils

The Yorkton Tribal Council:

• Provides INAC an updated assessment of the current planning and technical advisory services provided to member FN engaged in the CCBP initiative
  
  
• Ensures all financial transactions relating to CCBP support and capacity development are disclosed in a separate schedule of Revenues and Expenditures
  
  
• Administers contract with CEU
  
  
• Supports the community engagement, planning and reporting activities of FN
  
  
• Serves as the liaison between the respective Tribal Council executive committees and the regional JSC

Cities and Environment Unit (Dalhousie University):

• Provides training in community-based planning for community Plan Champions / Tribal Council employees through scheduled workshops in pilot communities and training sessions
  
  
• Provides training in the field through work in the pilot communities (learning by doing)
  
  
• Builds capacity and awareness in each community for the Plan Champion, associated Tribal Council, band administration, chief and council, and Planning Work Group
  
  
• Builds awareness of community-based planning at all levels: individual, community, Tribal Council and Federal Government through meetings, project newsletters and project website
  
  
• Identifies technological needs for planning at the local level
  
  
• Provides professional planning expertise, advice and guidance to the JSC and member FN communities
  
  
• Works alongside FN communities as active participants in plan development as well as implementation.

Federation of Saskatchewan Indian Nations (FSIN):

• In partnership with INAC Saskatchewan region, ensures coordination with existing and future governance capacity building processes (post‑FMM; Treaty Governance)
  
  
• Provides a forum through the Saskatchewan Legislative Assembly to create awareness and provide progress reports and communicate successes of CCBP initiative
  
  
• Appoints an FSIN representative to the JSC

Beneficiaries

The key beneficiaries of the CCBP are the FN of Saskatchewan. While the following fall under beneficiaries, they are also key stakeholders and project managers:

Tribal Councils:

• Responsible for the building of local CCBP capacity
  
  
• Designates employee to be intricately involved in the development of the plan
  
  
• Provides primary service to member FN seeking technical/ professional advice & assistance relating to the comprehensive planning process
  
  
• Advocates for change to ensure CCBP is supported with respect to the Tribal Council's annual budget and planning process
  
  
• Responsible for technical support to member FN engaged in the CCBP initiative
  
  
• Appoints Tribal Council representative to JSC
  
  
• Ensures adherence to the CCBP Terms and Conditions

The Community Contact

• Supervises and oversees the CCBP project, attends workshops, training sessions
  
  
• Develops the Plan with the Champion and ensures community participation

The Plan Champion

• Guides the planning process and builds momentum for the project through leading public participation events
  
  
• Works with the planning work group and analyzes information
  
  
• Ensures that work is completed in sequence and that all information gathered and created is shared with the entire community
  
  
• Reports project progress to Chief and Council

The Planning Work Group

• Gathers, assembles and presents information together with the Plan Champion,
  
  
• Ensures a broad base of involvement and support among community members in plan development and implementation,
  
  
• Works on the plan by attending meetings, creating maps, building models of the community and advancing the Plan ideas.

2.5 Program Resources

The CCBP pilot project has no A-base budget and is funded by allocations from other programs and agencies. Contributions come from the following INAC sources: the Professional and Institutional Development (P&ID) program, Capital program, Community Economic Development program and for the last three years, Health Canada.

	2005-2006		2006-2007		2007-2008		2008-2009		2009-2010		2010-2011
Category	Amount	% of Total	Amount	% of Total	Amount	% of Total	Amount	% of Total	Amount	% of Total	Amount	% of Total
CEU, Dalhousie	6,000	4.7	235,060	47.9	389,397	48.0	491,850	48.0	588,000	44.5	524,253	42.8
First Nations/ Tribal Councils	110,000	86.6	236,940	48.3	396,720	48.9	508,150	49.6	707,500	53.6	575,082	47.0
Joint Steering Committee Administration*.	11,000	8.7	19,000	3.9	25,000	3.1	25,000	2.4	25,000	1.9	25,000	2.0
Other (Diploma program)											100,000	8.2
Totals	127,000	100.0	491,000	100.0	811,117	100.0	1,025,000	100.0	1,320,500	100.0	1,224,335	100.0

(Source: INAC Saskatchewan region)

3. Evaluation Methodology

The following section focuses on the methodology that will be used to respond to the main questions the evaluation will strive to answer.

3.1 Evaluation Objectives

The evaluation will assess the extent to which the CCBP demonstrates relevance and performance in compliance with the Treasury Board Policy on Evaluation. The evaluation will also capture lessons and provide evidence-based recommendations with respect to CCBP.

3.2 Evaluation Scope

The evaluation will examine CCBP activities undertaken since its implementation in 2005-2006 up to 2010-2011.

3.3 Evaluation Issues

The evaluation will focus on the following issues:

Relevance

The evaluation will examine the relevance of the CCBP pilot project, including the extent to which it is consistent with federal roles and responsibilities and the needs and priorities of Saskatchewan FN communities.

Design

The evaluation will look at the extent to which the design of the CCBP project is appropriate in relation to the project's objectives, resources and context. It will also look at the adequacy and effectiveness of the governance structure.

Delivery

The evaluation will examine how CCBP was implemented, assessing such issues as the extent to which it was delivered in a timely, rational and efficient manner.

Effectiveness (Results/Success)

The evaluation will determine the extent to which the project's objectives have been achieved and identify the factors that have facilitated and/or limited the project's outcomes. It will also look at unintended results, both positive and negative, and the project's cumulative effects on individuals and participating communities. The evaluators will identify what worked well, what did not and why. In doing so, they will attempt to draw lessons that can be generalized across the region and across Canada.

Efficiency and Economy

Demonstration of efficiency and economy refers to the assessment of resource utilization (inputs) in relation to the production of outputs. It will examine whether the project used the most efficient means to obtain outcomes and whether there are practical alternatives.

3.4 Methodology

3.4.1 Data Sources

The evaluation findings and conclusions will be based on the analysis and triangulation of the following lines of evidence to respond objectively to the identified issues:

Literature Review:

The evaluators will conduct a review of relevant regional, national and international literature about the subject.

Document review:

Project documentation will be reviewed, including management information, contracts, project reporting and newsletters, budget information, community plans, CEU design-build booklets, and other related information.

Key informant interviews:

It is expected that a minimum of 24 and a maximum of 36 key informant interviews will be conducted, including, but not limited to:

• CEU (3)
  
  
• Yorkton Tribal Council (1)
  
  
• INAC Saskatchewan region managers, and officers (5)
  
  
• INAC headquarters (1)
  
  
• The Community Contacts (for each community, minimum 3, maximum 7)
  
  
• The Plan Champions (for each project, minimum 3, maximum 7)
  
  
• The Planning Work Group members (for each project, minimum 3, maximum 7)
  
  
• Chiefs and council members (3)
  
  
• Health Canada representative (1)
  
  
• FSIN representative (1)

Case Studies:

Four communities will serve as case studies for the collection of detailed qualitative and quantitative data on the project's outcomes on individuals and communities. The case studies will highlight best practices, lessons learned, key successes and how they were achieved as well as factors that were important in facilitating or hampering success.

Each of the case studies will incorporate information from the following lines of evidence:

• Document review, which will include, for example, the project proposal, the community plan, community profile data, CCBP project reporting
  
  
• Key informant interviews and focus groups, including with the Community Contacts, Plan Champions, Planning Work Group members, community members and leadership, and Tribal Council representatives.

3.5 Considerations, Strengths and Limitations of the Evaluation

As this is an evaluation requested by the Saskatchewan region, it is expected that cooperation between all the players will be high in relation to the provision of information required to perform the evaluation.

A limitation is the lack of a logic model and a performance measurement strategy. As a result, the available performance data are limited. The evaluators have drafted a logic model to assist with the evaluation and will attempt to reconstruct a baseline from available information.

4. Evaluation Matrix

The following Evaluation Matrix consists of the Evaluation Issues, Questions and Data Collection Methodologies.

Evaluation Matrix - Comprehensive Community-Based Planning (CCBP) Pilot Project, Saskatchewan

Issue/Research Questions	Indicators	Data/Methodology	Data Source
Relevance: Continued Need
1. Does the CCBP address a demonstrable need in Aboriginal communities?	Evidence of need for CCBP Degree to which CCBP is supporting FN community development (economic, social, politics)	Literature review Document review Key informant interviews Case studies	Internet, government policy documents, CCBP annual reports, academic papers Interviews with the community contacts, plan champions, planning work group members, Tribal Council representatives, CEU staff, INAC staff
Relevance: Alignment with government priorities
2. Are the objectives of CCBP consistent with departmental and government-wide priorities? Is it aligned with federal roles and responsibilities? Is this bottom-up, comprehensive planning model consistent with INAC policies and strategy?	Extent to which CCBP objectives are consistent with government and departmental priorities including INAC's Gender-based Analysis Policy, Sustainable Development Strategy, and new Community Development Framework	Literature review Document review Key informant interviews	Speech from the Throne, Budgets, Departmental Program Activity Architecture, Ministerial speeches and announcements, Gender-based Analysis Policy, Sustainable Development Strategy and draft documentation on Community Development Framework Interviews with INAC representatives (headquarters and Saskatchewan)
Relevance: Consistency with Federal roles and responsibilities
3. Does the CCBP duplicate or overlap with other departments or jurisdictions?	Information indicating duplication or overlap with other departments or jurisdictions	Document review Key informant interviews	Departmental mandate, policies, and other documents that discuss jurisdictional responsibilities Interviews with INAC; Health Canada, Tribal Council representatives
PERFORMANCE: Design
4.1 To what extent were the project objectives/outcomes clear to all major stakeholders and shared? Were there consistent, realistic expectations?	Extent to which project objectives are clearly outlined in project documentation Extent to which stakeholders had shared views of objectives and expectations	Document review Key informant interviews Case studies	Project documents, steering committee minutes Interviews with INAC staff, Tribal Council representatives, community contacts, plan champions, CEU staff
4.2 To what extent was the project design appropriate? Did the pilot project design help to improve planning processes? Was the model (stages) effective? Was there sufficient time for capacity development? Was there an equitable system in place for determining priorities and selecting projects at the community level?	Extent to which communities increased their planning capacity Perceptions of stakeholders as to the effectiveness of the staged approach to plan development implemented over three phases Perceptions of stakeholders regarding adequacy of the time available to develop planning capacity Extent to which the pilot design allowed for broad-based community involvement, including women, men, youth, elders & disadvantaged Extent to which the project design fostered equitable decision making	Literature review Document review Key informant interview Case studies	Evaluations Community plans, project reports, funding agreements Interviews with INAC staff, community contacts, plan champions, planning working group members, community leadership, CEU staff
4.3 To what extent was the governance structure appropriate?	Extent to which the JSC, Yorkton Tribal Council provided adequate direction and guidance Extent to JSC, Yorkton Tribal Council contributed to the achievement of results	Document review Key informant interviews Case studies	Project documents, steering committee minutes Interviews with INAC staff, Tribal Council representatives, community contacts, plan champions, CEU staff
PERFORMANCE: Delivery
5.1 Were there sufficient and appropriate resources and support to implement the project?	Level of INAC support Key informant perspectives on the sufficiency resources and INAC engagement	Literature review Document review Key informant interview Case studies	Evaluations Project reports, budget information Interviews with INAC staff, community contacts, plan champions, planning work group members, community leadership, CEU staff
5.2 Was there sufficient flexibility to allow the project to adapt to changing conditions/local contexts (language & culture)?	Evidence of adaptation	Document review Key informant interviews Case studies	Project reports, community plans Interviews with INAC staff, community contacts, plan champions, planning work group members, community leadership, CEU staff
PERFORMANCE: Effectiveness - Achievement of expected outcomes
6. What did the project achieve (expected and unexpected outcomes)? 6.1 To what extent is there increased awareness (on the part of pilot communities, INAC & Tribal Councils) of the importance of a comprehensive community-based plan?	# of stakeholders who indicate they have greater awareness of the importance of comprehensive community based planning	Document Review Key informant interviews Case Studies.	Project reports, community plans Interviews with INAC staff, Health Canada staff, community contacts, plan champions, planning working group members, community leadership and members, CEU staff
6.2 To what extent have individual and community stakeholders increased their capabilities regarding comprehensive community-based planning?	# of community members who actively participated in preparing the plans with no previous related experience Perceptions of stakeholders as to their increased capabilities	Document review Interviews Case Studies.	Project reports, community plans Interviews with INAC staff, Health Canada staff, Tribal Council representatives, community contacts, plan champions, planning work group members, community members and leadership, CEU staff
6.3 To what extent are the community plans being used by communities and INAC?	number of projects identified by the plans that have been implemented Extent to which communities are using the plans to set priorities/ allocate resources/ lever other programs/partnerships Perceptions of benefits to communities from implemented projects Extent to which the plans are being used by INAC and other government departments to inform program planning and funding decisions	Document review Key informant interviews Case studies	Project reports, community plans Interviews with INAC staff, Tribal Council representatives, community contacts, plan champions, planning work group members, community members and leadership, CEU staff
PERFORMANCE: Sustainability
7. To what extent will the benefits of CCBP continue after the pilot projects end? To what extent have communities taken ownership of their plans? Are there plans and resources in place to scale up this pilot initiative? To what extent, if at all, has INAC's planning approach and culture changed or been influenced by the CCBP?	# of pilot communities that have updated # of pilot communities that intend to update their plans independently Level of available resources Perceptions of stakeholders regarding the extent to which CCBP has led to change in approach/culture/ reduction in planning duplication	Key informant interviews Case studies	Interviews with INAC staff, Tribal Council representatives, community contacts, plan champions, planning working group members, community leadership, CEU staff
8. Have the CCBP pilots had unexpected outcomes, positive or negative?	Unexpected changes attributed by stakeholders to the pilot projects (e.g. changes in relationships within the community, with INAC, other programs and partners)	Key informant interviews Case Studies Document review	Project reports Interviews with INAC staff, Tribal Council representatives, community contacts, plan champions, planning working group members, community members and leadership, CEU staff
PERFORMANCE: Lessons
9. What mechanisms are in place to capture lessons and facilitate learning from the pilot initiatives?	Evidence of data collection data analysis and good reflective processes Evidence of information sharing with other communities	Document review Key informant interviews Case studies	Project reports Interviews with INAC staff, Tribal Council representatives, community contacts, plan champions, planning working group members, CEU staff
10. Are there lessons or alternatives that have emerged that could contribute to improved community planning?	Updates to guidance documents, procedures Scans of research, academic literature Opinions of project stakeholders	Key Informant Interviews Case studies	Interviews with INAC staff, Tribal Council representatives, community contacts, plan champions, planning working group members, CEU staff
PERFORMANCE: Economy and Efficiency - Resource utilization in relation to outputs
11. Are the most appropriate and efficient means being used to develop community plans	Cost per community Cost per community compared to other provinces	Literature review Document review Key informant interviews	Evaluations Financial data. project reports, INAC evaluations from other provinces Interviews with INAC BC officials, INAC Saskatchewan staff
12. Are there alternative models that could improve CCBP or reduce the cost?	Suggestions for alternative models/cost reductions	Literature review Key informant interviews Case studies	Academic articles Interviews with INAC staff, Tribal Council representatives, community contacts, plan champions, CEU staff

5. Project Management and Quality Control

The Evaluation, Performance Measurement and Review Branch (EPMRB) of INAC's Audit and Evaluation Sector (AES) will be the project authority for the CCBP pilot project evaluation. The evaluation will be performed by EPMRB with the assistance of Stiles Associates Inc.

Three members of the JSC representing INAC, Yorkton Tribal Council and the CEU were consulted in the preparation of this evaluation methodology report. They will review this draft report and once data collection is complete, they will participate in a presentation to validate the preliminary findings. They will also review the draft evaluation report.

6. Timeline

Task	Target Completion Date
1. Methodology Report	Mid-February 2011
2. Data collection Field research for case studies	March – April 2011 February 28-March 11, 2011
3. Presentation of Preliminary Findings	Mid-April
4. Draft Evaluation Report	Early May 2011
5. Final Report	May 31, 2011