Archived - Summative Evaluation of the Procurement Strategy for Aboriginal Business (PSAB)

Date: March 29, 2007

PDF Version (213 Kb, 54 Pages)

Table of Contents

• Executive Summary
• 1.0 Introduction
• 2.0 Methodology
• 3.0 Findings
• 4.0 Key observations
• Management Response / PSAB Action Plan

Executive Summary

Introduction

The Minister of Indian and Northern Affairs (INAC) announced the initiation of the Procurement Strategy for Aboriginal Business (PSAB) on March 29, 1996 in response to the under-representation of Aboriginal businesses among firms that were seeking and winning federal government contracts. The PSAB's main objective was "to assist Aboriginal business development by increasing Aboriginal business participation in the awarding of contracts by federal departments and agencies." It applies to all designated federal departments and agencies under the Financial Administration Act and includes the following three components:

• set-aside contracts for competition by Aboriginal businesses – it is mandatory to set aside any contracts for goods or services valued in excess of $5,000 that are destined primarily for Aboriginal populations, provided that operational requirements (best value, prudence and probity, and sound contracting management) can be assured, and contract authorities may also voluntarily decide to set aside other contracts for bidding by Aboriginal suppliers.
• means to encourage federal departments to increase their purchases from qualified Aboriginal firms – federal departments set targets for Aboriginal procurement and report annually on the fulfillment of these targets, and information sessions and other methods are used to promote the Strategy to federal contracting officers; in addition, each department has a PSAB coordinator that promotes the Strategy within the department.
• means to increase the awareness and capacity of Aboriginal firms to participate in federal contracting opportunities – various communication activities designed to raise awareness among Aboriginal suppliers of what federal departments buy, how they buy, and the manner in which they evaluate potential suppliers, are undertaken and joint ventures are encouraged.

Following the Treasury Board requirements, the summative evaluation of the PSAB was to be conducted by March 31, 2007. A work plan for the summative evaluation was developed in the summer of 2006, and the summative evaluation was undertaken from November 2006 to March 2007.

Methodology

This evaluation comprised several lines of enquiry: key informant interviews (with PSAB program staff and senior procurement officers); a survey of departmental PSAB coordinators; a survey of Aboriginal businesses that had received contracts under the PSAB; a survey of departmental material managers; and a document review/database assessment (comprising a review of program documentation and departmental performance reports, and analysis of data surrounding Aboriginal procurement provided by Public Works and Government Services Canada).

Key Findings

The summative evaluation explored issues surrounding the PSAB's rationale/relevance, design and delivery, impact/success, as well as cost-effectiveness, lessons learned, and alternatives to the PSAB.

Rationale

Government stakeholders who participated in the evaluation generally continue to support the PSAB's rationale and objectives; however, the response rate among material managers was low, suggesting that support may not be as widespread as was suggested by the data obtained. There is more support for increasing opportunities for Aboriginal firms in other markets, and for increasing the number of Aboriginal firms competing for federal contracts, than for goals such as increasing the value of contracts awarded to Aboriginal firms, which could, in part, be a reflection of the fact that the number and value of contracts awarded to Aboriginal firms has increased dramatically since the inception of the PSAB, making for a different policy context than when the Strategy was implemented over a decade ago. Interviews with PSAB program staff suggest a desire to move beyond the PSAB by helping Aboriginal firms that have benefited from the Strategy to continue to grow without relying solely on contracts set aside under PSAB.

Design and Delivery

A number of stakeholders suggested that there are not adequate means of enforcing compliance with the PSAB, given that there are no institutionalized consequences for departments that fail to set aside contracts under the PSAB or that fail to achieve their performance targets. The data indicate there are discrepancies across departments in terms of the amount of time devoted to the PSAB, the use of PSAB set-asides, and the perceived level of awareness of the PSAB required among material managers.

Some stakeholders also indicated that the eligibility requirements of the PSAB are unclear – particularly the Aboriginal content requirements – and that they are in need of further clarification. Survey and interview data suggest that many material managers and senior procurement officers are not highly familiar with the requirements and/or are unaware that the requirements had recently been clarified.

While it is clear that only a minority of audited firms are found to be non-compliant, the consequences of non-compliance are unclear and are not institutionalized, suggesting that potential exists to strengthen the enforcement and transparency surrounding the audit process.

Impact/Success

While it is evident that both the number and the dollar value of contracts awarded to Aboriginal firms has increased since the inception of the PSAB, there is little data that speaks to the achievement of other expected impacts, particularly the expected indirect impacts. Aboriginal businesses indicated that the PSAB had a positive impact on such things as their ability to compete for federal contracts and also the number and value of contracts awarded to their firm. They indicated that the PSAB had less of an impact upon investment in their firms as well as their ability to develop product lines to meet demands. Many were unable to assess the impact of the PSAB upon Aboriginal businesses and Aboriginal communities, in general.

PSAB program staff, in particular, were concerned with the lack of impact data available. They indicated that additional data should be systematically collected to assess the PSAB's impact, including the number of jobs, training and apprenticeship opportunities created; as well as whether, and to what extent businesses have been able to expand; the number of employees who have been hired who were previously on social assistance; whether, and to what extent, sales and profit margins increased; and whether the businesses can now access opportunities in the international marketplace. The survey data suggests that in a minority of firms, there has been some degree of job creation as a result of the PSAB.

Lessons Learned and Future Directions

Survey data from material managers and PSAB coordinator indicates that finding Aboriginal firms with the appropriate capacity to undertake particular government contracts remains a significant challenge. Interview and survey data suggest this is due to the lack of an accessible list of prequalified suppliers, and gaps in capacity in particular fields, such as the medical, architectural, engineering, and surveying fields, and the skilled trades.

PSAB program staff recognized a need to build Aboriginal capacity in certain areas and see the implementation of an Aboriginal benefits strategy as being important in this regard. Under such a strategy, an Aboriginal benefits requirement would be included on large contracts whereby contractors would be required to demonstrate some commitment to benefiting Aboriginal people in order to be awarded these contracts, through, for example, the establishment of a scholarship program, the creation of jobs for Aboriginal people, or a commitment to acquire materials and supplies from Aboriginal vendors. Program staff believe the Aboriginal benefits strategy should be implemented in conjunction with, rather than as a replacement to, the PSAB, as they see the PSAB as an important means through which new and emerging Aboriginal businesses can gain experience. A Memorandum to Cabinet (MC) would be required in order to make substantive changes to the PSAB, such as the inclusion of an Aboriginal benefits requirement.

There was a strong consensus among all parties that there is a need to raise awareness of the PSAB as well as to actively promote its use.

1.0 Introduction

The Procurement Strategy for Aboriginal Business (PSAB) was put in place in 1996. Following the Treasury Board requirements, a first evaluation of the Strategy was conducted in 2001, with the summative evaluation of the PSAB to be conducted by March 31, 2007.  In the summer of 2006, Indian and Northern Affairs Canada (INAC) contracted PRA Inc. to develop a work plan to conduct the summative evaluation of the PSAB. The plan was developed based upon a review of documentation and interviews with PSAB program staff, as well as representatives from Public Works and Government Services Canada (PWGSC), Consulting and Audit Canada (CAC), and the Treasury Board Secretariat (TBS).

In November 2006, INAC contracted PRA Inc. to conduct the summative evaluation of the PSAB using an amended version of the methodology that had been presented in the evaluation work plan. This report presents the findings of the evaluation surrounding the PSAB's rationale/relevance, design and delivery, impact/success, as well as cost-effectiveness, lessons learned, and alternatives to the PSAB.

1.1 Description of the Procurement Strategy for Aboriginal Business (PSAB)

The Minister of Indian and Northern Affairs (INAC) announced the initiation of the Procurement Strategy for Aboriginal Business (PSAB) on March 29, 1996. [Note 1]. The Strategy was developed in response to the under-representation of Aboriginal businesses among firms that were seeking and winning federal government contracts. [Note 2]. Its main objective was "to assist Aboriginal business development by increasing Aboriginal business participation in the awarding of contracts by federal departments and agencies." [Note 3].

The PSAB applies to "all federal departments and agencies designated as departments for the purposes of the Financial Administration Act" [Note 4]. and includes the following three activities aimed at promoting Aboriginal business participation in government procurement:

• set-aside contracts for competition by Aboriginal businesses;
• means to encourage federal departments to increase their purchases from qualified Aboriginal firms; and
• means to increase the awareness and capacity of Aboriginal firms to participate in federal contracting opportunities.

1.1.1 Set-aside Contracts

The North American Free Trade Agreement (NAFTA) and the World Trade Organization Agreement on Government Procurement (WTO-AGP) provide for procurements to be "set aside" for minority and small businesses. Procurements set aside for Aboriginal businesses are therefore excluded from the provisions of these two international trade agreements. [Note 5] Under the PSAB, federal departments set aside contracts exclusively for Aboriginal businesses, using two devices:

• Mandatory set-asides - Any contracts for goods or services valued in excess of $5,000 destined primarily for Aboriginal populations are reserved exclusively for Aboriginal suppliers provided operational requirements, namely best value, prudence and probity, and sound contracting management can be assured.
• Voluntary set-asides - Contract authorities may also voluntarily decide to set aside other contracts for bidding by Aboriginal suppliers (including contracts valued at less than $5,000 destined primarily for Aboriginal populations). Decisions to set aside voluntarily "should be founded on knowledge and expertise of the Aboriginal supplier community, meeting departmental objectives, best business practices, and the nature of the requirement." [Note 6]

1.1.2 Means to Encourage Federal Departments to use Aboriginal Suppliers

The PSAB encourages federal departments to identify targets for Aboriginal procurement and to report annually on the fulfillment of these targets.  Information sessions, communications, and other methods are used to promote the strategy to federal contracting officers and to inform contracting officers within the department about the capacities of Aboriginal firms. To encourage government departments and agencies to do business with Aboriginal suppliers and be receptive to Aboriginal suppliers' marketing efforts, departments and agencies with annual contracting budgets in excess of $1.0 million "must develop multi-year performance objectives for contracting with Aboriginal businesses and develop related reporting mechanisms." [Note 7] This includes performance objectives for "issuing contracts to Aboriginal suppliers, performing Aboriginal supplier development activities, and including Aboriginal suppliers on the source lists." [Note 8] Each federal department has a PSAB coordinator who promotes the initiative within the department and works with management to establish and report upon annual targets for Aboriginal procurement.

1.1.3 Means to Increase the Awareness and Capacity of Aboriginal Firms to Participate in Federal Contracting Opportunities

Federal departments and agencies undertake various communications activities to raise awareness among Aboriginal suppliers of what they buy, who makes buying decisions, what they look for when they buy, how they buy, and the manner in which they evaluate potential suppliers. [Note 9] Aboriginal suppliers are encouraged to enter into joint ventures with other Aboriginal or non-Aboriginal suppliers to bid on government contracts; such joint ventures qualify as Aboriginal suppliers under the PSAB if the Aboriginal business/businesses have at least 51% ownership and control of the joint venture. [Note 10]

1.2 Organization of the Report

This report includes four main sections, including this introduction. Section 2 details the methodology that was used to conduct the evaluation, including a discussion of methodological limitations, and presents the evaluation matrix that was used to guide the research. Section 3 presents key findings for each of the questions presented in the evaluation matrix, and Section 4 summarizes the key conclusions of the evaluation.

2.0 Methodology

This section will outline the lines of enquiry that the evaluation comprised, discuss any methodological limitations of the data, and present the evaluation questions and the indicators that were used to assess them.

2.1 Lines of Enquiry

This evaluation comprised several lines of enquiry: key informant interviews, a survey of PSAB coordinators, a survey of Aboriginal businesses that had received contracts under the PSAB, a survey of departmental material managers, and a document review/database assessment.

2.1.1 Key Informant Interviews

Two groups of key informants were interviewed – PSAB program staff (n=7) and senior procurement officers (n=13). Separate interview guides were constructed for each of these groups to ensure the relevance of the questions, although the content and themes of the interviews overlapped considerably.

Key PSAB program staff (n=7) were interviewed either in-person or by phone to gain insight into the PSAB's rationale/relevance, design and delivery, impact/success, as well as cost-effectiveness, lessons learned, and alternatives to the PSAB.

Senior procurement officers (n=13) from a variety of government departments of varying sizes and structures, including INAC, were interviewed by telephone to gain insight into the evaluation issues from the perspective of different departments. INAC was responsible for selecting the departments from which senior procurement officers would be interviewed and acquiring an appropriate contact name via the PSAB coordinator. PRA then followed up with these contacts to schedule the interviews; some of the senior procurement officers who had been identified declined to participate or did not return any of numerous calls placed to schedule their interviews. As a result, the actual number of interviews conducted was 13, rather than 15, as had originally been intended.  Key findings from the interviews have been integrated throughout this final report.

2.1.2 Survey of PSAB Coordinators

All departmental PSAB coordinators (n=54) listed on the PSAB website received questionnaires to collect information on their department's PSAB-related activities, as well as to ascertain their opinions regarding the rationale/relevance, design and delivery, and impact and success of the Strategy.

The questionnaire was pre-tested with three respondents in order to ensure the questions were clear and appropriate to the respondents to be surveyed. Bilingual paper copies of the questionnaire were mailed to all PSAB coordinators in mid-December. The surveys were mailed with an addressed postage paid envelope in which respondents could return their completed questionnaires. The paper copy of the survey also gave respondents the option of receiving an electronic version of the questionnaire that could be completed on the computer. In early January, PSAB coordinators that had yet to return their questionnaires were sent e-mail reminders. A second e-mail reminder was sent out approximately two weeks later (the fourth week of January).

Pre-tests included, 28 questionnaires were returned, for a response rate of 52%. The largest proportion of respondents (43%) indicated they had been the PSAB coordinator for their department for 5 years or more, whereas 25% indicated they had been their department's PSAB coordinator less than a year. Respondents most commonly indicated that they worked in sections or divisions responsible for procurement and contracting. Just over two thirds of respondents (68%) indicated that their divisions or sections undertake the following activities to support Aboriginal procurement: promoting Aboriginal procurement objectives, overseeing contract procedures, and monitoring procurement contracts. A significant proportion indicated their section or division purchases goods and services (61%) and maintains and uses Aboriginal supplier lists (50%), while a minority (18%) specified that they meet with Aboriginal business suppliers.

2.1.3 Survey of Businesses that Received Contracts Under the PSAB

PWGSC provided PRA Inc. with a database of all contracts awarded by PWGSC to Aboriginal firms since the inception of the PSAB (1996) to October 2006. [Note 11] Contracts awarded under the PSAB were extracted from this sample using the trade agreement description indicator. Approximately 500 unique vendors were identified; however, although the business name, address, and telephone numbers were included in the database, no contacts names were provided. Given that addressing business questionnaires to particular titles (e.g., "the owner") rather than to named individuals tends to result in a low response rate, as well as to address the possibility that some vendors in the sample may have moved or may no longer be in business, PRA placed calls to these vendors in November 2006 to acquire contact names, confirm mailing addresses, and advise vendors to expect a questionnaire in the mail.

From an initial sample of over 500 vendors that had received contracts under the PSAB, telephone calls to each to acquire the appropriate contact names and confirm contact information revealed that many of the business phone numbers were no longer in service. Attempts were made to look-up businesses to identify their new phone numbers, if they were available, but in many cases, no current number could be identified. The sample therefore decreased to 310, though contact names could not be acquired for all firms.

Surveys were sent to the businesses in the sample via regular mail and mail reminders followed approximately two weeks after. In addition, follow-up phone calls ensured that the questionnaires had been received and were used to encourage contacts to complete and return the questionnaires. During these phone calls, businesses were also given the opportunity to receive the questionnaire by either fax or as an electronic version via e-mail. A second follow-up call was placed a couple of weeks later to remind businesses to complete the questionnaire, at which point businesses were also given the opportunity to complete the questionnaire over the phone. Due to these repeated efforts, 92 questionnaires were completed, for a response rate of approximately 30% - which is considered quite a good response rate for a business survey.

The median number of years in business among survey respondents was 9. Over half of businesses (57%) indicated their businesses were 100% Aboriginal owned, while 26% indicated that they were 51% Aboriginal owned. A variety of sectors were represented, through the most predominant were construction (28%), manufacturing (20%), government service (15%), health and social service (14%), business service (14%), retail and wholesale (13%), and other service industries (13%). [Note 12] The median total number of full time employees was 6, with nearly one fifth (18%) indicating that they had one employee or less. In the most recent fiscal year, nearly one third (29%) indicated their revenue had $250,000 or less while 15% reported revenues of more than $5,000,000. Within the last fiscal year, the median percentage of the total value of sales to federal government was 43%, with just over 1 in 10 (12%) indicating that the federal government had accounted for 100% of the value of their sales.

2.1.4 Survey of Departmental Material Managers

The sample of departmental material managers and their contact information was acquired from the Contracts Canada website. The questionnaire was pre-tested with two material managers to ensure the questions and wording was appropriate. In the second week of December 2006, the material managers in the sample (n=666) were then sent the questionnaire both by regular mail and e-mail, giving them the option to complete the questionnaire in either electronic or hard copy format. An e-mail reminder was sent in early January 2007, followed by a second e-mail reminder toward the end of January 2007.

A small proportion of the contacts from the original sample had moved or quit (n=17 or 3%) or indicated they were not qualified to complete the questionnaire (n=11 or 2%). A total of 144 questionnaires were returned, for a response rate of 23% of the remaining sample. Over half of the respondents (54%) indicated they had been departmental material managers for their current departments for 5 years or more, whereas nearly one third (29%) had between 1 and 5 years' experience as material managers for their departments, and 15% had less than a years' experience.

2.1.5 Database Assessment/Document Review

PWGSC provided a database generated by their Acquisition Information System (AIS) of all contracts (both set-asides and open competition) awarded by the department to Aboriginal firms since the inception of the PSAB in 1996. [Note 13] The database also included the names, locations, industry sectors, and sizes (number of employees) of the firms awarded these contracts, the dates and dollar values of the contracts, as well as the department on whose behalf each contract was awarded. This data was analyzed to generate a profile of the PSAB contracts and the firms receiving contracts under the PSAB and a comparison between PSAB and non-PSAB contracting activity.  Key findings from the database assessment have been integrated throughout this final report.

INAC provided a select number of departmental PSAB performance reports and data surrounding audits conducted by CAC for review. Findings from the document review have also been incorporated into this report.

2.2 Limitations of the Data

While the evaluation offers insight into the perceived legitimacy of the rationale for the PSAB, as well as its design and delivery and its impacts, it is important to understand the limitations of this data. Every effort was made to collect tangible factual data, however, there is limited performance measurement data being collected to assess the impact of the PSAB. This challenge arose during the first evaluation of the PSAB, which yielded recommendations for improving reporting. Standardized reporting has yet to be implemented, however, as it was decided to postpone such activity until after the government-wide review on procurement that is being undertaken was completed. As such, since raw data from all government departments could not be obtained, the evaluation relies upon data from PWGSC, as well as the data collected within the surveys of PSAB coordinators, material managers, and businesses, much of which are perceptions or opinions only. It is important to remember that opinion data, while valuable, is not always a reliable assessment of actual impacts.

With regard to the Aboriginal business survey, it should be noted that the sample was generated from firms that had been awarded contracts by PWGSC (either directly or on behalf of another department) only, since data regarding contracts awarded directly by other departments was not available. Furthermore, the sample included standing offers that had not been called up, and a number of the businesses included therein were no longer in business. Some of the businesses contacted were unaware of the PSAB, in spite of records showing they had received at least one PSAB contract. Some of the factors that may account for this include turnover and succession, as well as difficulty recollecting details of contracts that may have been awarded several years ago.

The database assessment suffers from these same limitations and it is important to note that the firms that are denoted as Aboriginal firms have been self-identified, but not necessarily verified, as such.

Methodologically, to assess the impact and reach of the impact, it would have been useful to have conducted a general survey of Aboriginal businesses in Canada, not just those that reported having received such contracts, to assess awareness and opinion of the Strategy among Aboriginal businesses in general. This may have offered additional insights into potential gaps in the Strategy, though it may not have been possible to construct a reliable sample frame of Aboriginal businesses. In addition, some qualitative research such as focus groups/work groups or interviews with Aboriginal businesses may have offered more in-depth insights than quantitative survey data can provide.

The response rate for the survey of departmental material managers was rather low (23%), and though the response rate among PSAB coordinators was higher (52%), nearly half of departmental PSAB coordinators did not respond to the survey in spite of repeated follow-up attempts. The original evaluation methodology called for ADM-level involvement as a means of enhancing these surveys' response rates. Originally, an e-mail from an ADM from INAC was to be sent directly to those who were being surveyed in order to encourage material managers and PSAB coordinators to complete and return the questionnaires that they received directly from PRA; it was later determined that it would be inappropriate for an ADM from INAC to directly address subordinates in other departments, and therefore, the strategy was modified as follows:

The ADM of INAC was to correspond with other departmental ADMs, who, it was hoped, would then encourage or remind the material managers and PSAB coordinators, within their respective departments, to complete the questionnaires that had been sent to them. Unfortunately, due, in part, to recent turnover within INAC, such correspondence could not be approved and sent out prior to the close of the survey. Had the evaluation contact strategy been implemented as originally envisioned, with more buy-in from the ADM-level, the response rate may have been improved.

2.3 Key Evaluation Issues

The evaluation is intended to address the following key issues:

• the rationale/relevance and appropriateness of the PSAB, its components, its objective, and its expected impacts
• the Strategy's design and delivery, including the appropriateness, clarity, and effectiveness of the PSAB's guidelines and eligibility criteria and the means of assessing the eligibility of Aboriginal firms and the ensuring government departments comply with the PSAB
• the impact/success of the Strategy, including the extent to which the PSAB has achieved its expected direct and indirect impacts, whether any unintended outcomes have been observed, and the Strategy's reach
• cost-effectiveness and lessons learned, including the perceived strengths and weaknesses of the PSAB and alternatives or future directions for promoting Aboriginal business development.

3.0 Findings

This section presents the findings of the evaluation. Results appear within four sections corresponding to the main evaluation issues: rationale and relevance; design and delivery; impact and success; and cost effectiveness, lessons learned, and alternatives. Each section will conclude with a synthesized summary of findings.

3.1 Rationale/Relevance

This section will present the findings related to the rationale and relevance of the PSAB.

3.1.1 Rationale for the PSAB and Each of its Components

This section will discuss the rationale for the PSAB and its components and their perceived relevance, drawing from the documentation available, as well as the interviews with PSAB program staff and the surveys of PSAB coordinators. The purpose was to clarify the rationale for the PSAB and to assess the extent to which it remains relevant more than a decade after the inception of the Strategy.

"PSAB – Guidelines for Buyers/Government Officials", revised in December 2005, indicates that the overall rationale for the PSAB was to increase contracting between the federal government and Aboriginal businesses. In establishing the PSAB, the federal government recognized that the number of Aboriginal businesses, as well as the number of qualified Aboriginal college/university graduates, had increased, and that the Aboriginal business community had the capability and the potential to play a much larger role in federal procurement. The government also recognized that "many business interests, especially small business, perceive the federal procurement process as being one of the main vehicles to allow them to achieve a higher level of economic viability and profitability." [Note 14] Consequently, it is apparent that enhancing the profitability of Aboriginal businesses, particularly small Aboriginal businesses, is also part of the Strategy's rationale. The rationale for the promotion of joint ventures is that "while many Aboriginal businesses may not yet be in a position to bid on very large projects, they can enhance their competitive edge by building and expanding partnerships and strategic alliances with others". [Note 15] It is expected that through the contracting mechanisms under the PSAB, socio-economic benefits to Aboriginal firms will be accrued and that more Aboriginal businesses will " become competitive within the mainstream Canadian economy". [Note 16]  The PSAB guidelines indicate that "PSAB is based on the premise of establishing Aboriginal business capacity, of value added by the Aboriginal business and demonstrated benefits to the Aboriginal community". [Note 17] The expected impacts/benefits of the PSAB will be elaborated upon in Section 3.1.2.

To clarify the current theoretical underpinnings of the Strategy and each of its components, PSAB program staff were asked to identify the rationale for the PSAB as a whole, as well as the rationale for set-asides, supplier development and information activities, departmental performance objectives, and mechanisms to encourage joint ventures between Aboriginal and non-Aboriginal businesses:

• Program staff indicated the rationale for the PSAB is to help Aboriginal businesses access federal contracting opportunities, thereby enhancing Aboriginal business development.
• Program staff viewed set-asides as a means through which Aboriginal businesses can gain experience, grow, and ultimately develop the capacity to win federal contracts under open competition. They emphasized that set-asides are particularly important for new Aboriginal businesses that may have difficulty winning contracts under open competition.
• Program staff indicated the rationale for supplier development and information activities is to help Aboriginal businesses learn how to access federal government contracting opportunities and market their services, and though they indicated that a strong rationale for supplier development and information activities remains, some noted the need to help Aboriginal firms grow without relying upon set-asides.
• Program staff indicated the rationale for departmental performance objectives is to encourage government departments to include Aboriginal suppliers in their supply chain, though a couple of program staff questioned the validity of this rationale, given the lack of means of enforcing compliance.
• Program staff indicated the rationale for joint ventures is to help Aboriginal businesses grow and become more competitive by allowing Aboriginal businesses to partner with other businesses in order to bid on and win larger contracts they may not have had the capacity to complete alone. Program staff indicated the expectation is that this will allow Aboriginal businesses to learn, expand, and increase their networks, and eventually, become able to bid on these larger contracts on their own.

Similarly to PSAB program staff, the survey results suggest that PSAB coordinators also see a strong rationale for PSAB and its components. As shown in Table 1, over three quarters of PSAB coordinators expressed agreement that the rationale for the PSAB remains valid, that departmental performance objectives should continue to include Aboriginal procurement, and that the set-asides are a valid way of supporting Aboriginal economic development. Overall, there was slightly less support for the encouragement of joint ventures and subcontracting arrangements, though most nevertheless agreed that a strong rationale remains. It appears that most PSAB coordinators also continue to see a rationale for supplier development activities, given that two thirds (64%) expressed disagreement that supplier development activities are less important now than five years ago.

3.1.2 Perceived Relevance and Appropriateness of Objectives and Expected Impacts

The objective and expected direct and indirect impacts of the PSAB are presented in Table 2. Interviews with PSAB program staff and senior procurement officers, as well as the surveys of PSAB coordinators, material managers, and Aboriginal businesses, were used to assess the extent to which the objectives and expected impacts of the Strategy are seen by these stakeholders to be relevant and appropriate.

Both PSAB program staff and senior procurement officers were asked whether the objectives and expected impacts of the PSAB, both direct and indirect, as shown in Table 2, are still relevant and whether any changes to the objectives and expected impacts are required. All program staff stated that the objectives and expected impacts of the PSAB remain relevant, though a few indicated that the expectations are high and go well beyond what the PSAB has the capacity to deliver. Senior procurement officers generally agreed that the objectives and expected impacts of the PSAB remain relevant, though a few raised concerns surrounding the implications of increasing partnerships between Aboriginal and non-Aboriginal businesses, noting that not all partnerships are legitimate, and limited benefits accrue to Aboriginal businesses if little to none of the work is performed by Aboriginal workers.

Figure 1 presents the levels of agreement of PSAB coordinators (denoted by PC) and material managers (denoted by MM) with statements regarding the relevance and appropriateness of PSAB's objectives and expected impacts. Both PSAB coordinators and material managers were most inclined to express agreement with the need to increase opportunities for Aboriginal firms in other markets and increase the number of Aboriginal firms competing for federal government contracts, while they were least inclined to express agreement that increasing the number of partnerships between Aboriginal and non-Aboriginal firms would increase the value from set-asides and that the value of contracts awarded by their departments to Aboriginal firms should increase. Support for the objectives was higher among PSAB coordinators than among material managers, and less than half of material managers expressed agreement that their department should increase the number or the value of contracts awarded to Aboriginal firms; approximately one quarter expressed disagreement with each of these statements, while a similar proportion did not respond or did not know.

Coordinators' and Managers' Agreement with PSAB's Objective and Expected Impacts

Text desctiption - Coordinators' and Managers' Agreement with PSAB's Objective and Expected Impacts

This figure shows a graph comparing the levels of agreement of PSAB Coordinators and Material Managers with statements regarding the relevance and appropriateness of PSAB's objectives and expected impacts.

The following five statements are along the vertical y axis:

• "there is a need to increase opportunities for Aboriginal firms in other markets"
• "the number of Aboriginal firms competing for federal government contracts should increase"
• "in this department, we need to increase the number of contracts to Aboriginal firms"
• "increasing the number of partnerships between Aboriginal and non-Aboriginal businesses would increase the value this department gets from set-asides"
• "the value of contracts this department awards to Aboriginal firms should increase"

The percentage rates, in 10% increments, for "strongly agree/agree" responses are along the horizontal x axis. Grouped with each statement are two bars, the top one representing PSAB coordinator responses and the bottom one representing Material Manager responses.

For the statement "there is a need to increase opportunities for Aboriginal firms in other markets," 79% of the 28 PSAB coordinators who participated and 65% of the 144 Material Managers who participated indicated that they "strongly agree/agree." 

For the statement "the number of Aboriginal firms competing for federal government contracts should increase," 71% of the 28 PSAB coordinators who participated, and 66% of the 144 Material Managers who participated indicated that they "strongly agree/agree."

With regards to the statement "in this department, we need to increase the number of contracts to Aboriginal firms," 68% of the 28 PSAB coordinators who participated indicated that they "strongly agree/agree" with this statement, and 48% of the 144 Material Managers who participated indicated that they "strongly agree/agree" with this statement. 

The statement "increasing the number of partnerships between Aboriginal and non-Aboriginal businesses would increase the value this department gets from set-asides" had 57% of the 28 PSAB coordinators who participated indicating that they "strongly agree/agree," and 51% of the 144 Material Managers who participated indicating that they also "strongly agree/agree." 

Finally for the statement "the value of contracts this department awards to Aboriginal firms should increase" 50% of the 28 PSAB coordinators who participated and 43% of the 144 Material Managers who participated indicated that they "strongly agree/agree" with this statement.

The expected indirect impacts of the PSAB imply that the positive impacts of the Strategy are expected to flow from the businesses to the communities. To assess the extent to which Aboriginal business success can be expected to benefit their communities, Aboriginal businesses were asked to indicate to what extent the success/profitability of an Aboriginal business affects the overall prosperity of the community in which it operates. Three quarters of business respondents (76%) indicated that the success or profitability of an Aboriginal business affects the overall prosperity of the community in which it operates at least to some extent, including nearly four tenths (37%) who indicated that Aboriginal business success affects community prosperity to a great extent.

3.1.3 Perceived Role of PSAB in Enhancing Business Success

To clearly identify the logic of the Strategy and its perceived role in enhancing Aboriginal business success, PSAB program staff and senior procurement officers were asked what they perceived to be the Strategy's role in enhancing Aboriginal business success. PSAB program officers indicated PSAB is intended to provide opportunities for Aboriginal businesses to compete for and acquire federal contracts. A few indicated that they hope with time Aboriginal firms will move beyond PSAB and become successful in federal contracting outside the PSAB. Senior procurement officers indicated that trade shows and the availability of PSAB program staff to whom Aboriginal businesses can refer, to enhance Aboriginal business success. Some, however, indicated that the Strategy will only be as successful as the level of awareness surrounding it, and that there is a need to provide more education to departmental officials, particularly managers and procurement officers. Like PSAB program staff, some senior procurement officers said the PSAB had played an important role by allowing Aboriginal businesses to compete and acquire federal contracting opportunities, which they had not previously been accessing, thereby allowing them to gain experience. A couple of officers, however, indicated that while the PSAB has significantly impacted upon some Aboriginal businesses, this has not been the case for all, particularly given some of the ambiguities surrounding Aboriginal content and joint ventures.

In order to facilitate the identification of appropriate indicators to assess the success of the PSAB, PSAB program personnel were asked to indicate what direct and indirect outcomes one could expect to see that would demonstrate the success of the PSAB. They indicated that direct expected outcomes would include more contracts awarded to Aboriginal suppliers and more Aboriginal businesses in operation. Indirect expected impacts would include Aboriginal suppliers providing community leadership and serving as role models for Aboriginal youth, as well as more jobs, increased job sustainability, and increased capacity. To assess the success of the PSAB, it was indicated that the number of jobs and training and apprenticeship opportunities created for Aboriginal people should be measured, as well as whether, and to what extent, businesses have been able to expand, the number of employees hired who were previously on social assistance, whether, and to what extent, sales and profit margins increased, and whether the business was able to gain access opportunities in the international marketplace – indicators that are not currently being collected and reported upon.

3.2 Design/Delivery

The following section addresses issues surrounding the design and delivery of the PSAB, including expectations of government departments, compliance mechanisms, and eligibility criteria.

3.2.1 Expectations of Government Departments in Supporting the PSAB

To identify the expectations of government departments in supporting the PSAB, documentation surrounding the PSAB was reviewed. PSAB program staff and senior procurement officers were asked to describe what federal departments are expected to do in support of the PSAB, and how these expectations have changed over time to assess the clarity and level of awareness surrounding these expectations.

The PSAB policy, available on INAC's PSAB website, includes the following statements surrounding the expectations of federal government departments under the PSAB:

• "all departments and agencies shall initiate or participate in supplier development activities aimed specifically at Aboriginal businesses"
• "It is essential that contract administrators are familiar with the supply and qualifications of Aboriginal businesses, especially in regional situations. Federal departments are expected to lead by example and practice in contracting with qualified Aboriginal businesses."
• "All departments and agencies are authorized and encouraged to voluntarily set aside other procurements under the set aside program for Aboriginal business where practical and cost effective."
• "Each department and agency with an annual contracting budget in excess of $1.0 million shall develop multiyear performance objectives for contracting with Aboriginal businesses."
• "Departments must retain for review and audit details of procurements in order to monitor the success of this policy and to prepare appropriate reports to the Treasury Board." [Note 18]

PSAB program staff and senior procurement officers, asked what government departments are expected to do in support of the PSAB and how these expectations have changed over time, generally identified expectations in keeping with the documentation surrounding the Strategy. Program staff indicated that government departments are expected to be aware of and to make use of the PSAB, to set targets for Aboriginal procurement, to advocate for the Aboriginal supplier community, and to actively seek out contracting opportunities with them. Similarly, senior procurement officers indicated that they are expected to adhere to the PSAB, to set and report upon targets for Aboriginal procurement, and to promote the Strategy and Aboriginal businesses in general. Some senior procurement officers indicated that their departments were proactive in setting aside contracts for Aboriginal firms, whereas others indicated they did not make use of set-asides since they did not apply to their departments' contracts and/or because they were able to meet their targets without doing so. Both program staff and senior procurement officers generally indicated that expectations have remained unchanged, though some program officers noted not all government departments have met these expectations.

3.2.2 Compliance Mechanisms

Interviews with PSAB program staff and senior procurement officers, as well as the survey of PSAB coordinators, were used to identify what mechanisms are in place to ensure government departments comply with the PSAB's requirements, to assess the perceived effectiveness of these mechanisms, and to solicit suggestions for improving these mechanisms.

PSAB program staff and senior procurement officers indicated that the following help ensure compliance with the PSAB:

• The PSAB is part of Treasury Board Contracting Policy.
• A member of the PSAB team monitors MERX to ensure appropriate contracts are being set aside, and contacts contracting officers to inform them when contracts should be set-aside.
• Government departments establish annual targets for Aboriginal procurement and report upon their progress with year-end reports provided to Deputy Ministers.
• PSAB staff follow up with departments that have not met their targets to encourage them to do more to meet the targets the following year or to find out why they have not met their targets.

While many senior procurement officers indicated these means were effective in ensuring compliance, program staff indicated a need to strengthen compliance mechanisms. Program staff suggested tying achievement of the targets to performance pay and indicated there should be more commitment on the part of Treasury Board Secretariat, ADMs, and the Auditor General to ensure compliance. Similarly, a few senior procurement officers commented that compliance could be improved through greater political emphasis, more buy-in and commitment from upper management, and more follow-up and monitoring of the objectives.

Less than half of PSAB coordinators (43%) indicated that there are effective mechanisms that ensure compliance with the PSAB; 39% indicated there were no effective mechanisms and 18% did not know. Those who indicated there were no effective compliance mechanisms in place were asked what needs to be done to increase compliance. Respondents indicated that departments have no incentive to comply, that there were no penalties for non-compliance and that more training for new and existing staff was needed.

Asked what mechanisms are in place to encourage their departments to meet their performance objectives, many PSAB coordinators (29%) did not know or did not respond. Those who did respond primarily indicated that achieving the performance objectives had been made a priority and that there were contract stipulations and PWGSC tools in place (see Table 3).

In sum, there do not appear to be any coercive means of ensuring government departments comply with the PSAB or institutionalized penalties for government departments that fail to set aside contracts when appropriate or that fail to meet their performance objectives. Stakeholder response suggests the need for sanctions for non-compliance and also implies that offering more PSAB-related training and fostering commitment to the Strategy among top-level departmental personnel's might also enhance compliance.

The evaluation of the PSAB completed in 2002 identified the lack of means of enforcing compliance with the PSAB as an issue and recommended enhancing enforcement of the Strategy. Following this evaluation, an Interdepartmental Working Group was formed to assist in revitalizing the PSAB and to explore the evaluation recommendations. The feasibility of "introducing performance pay penalties as an incentive to senior officials in promulgating the use of the PSAB among departmental procurement personnel" was explored by the Working Group, as well as an ADM-level Steering Committee, but was ultimately rejected in favour of a focus on a government-wide communications strategy. [Note 19]

3.2.3 Implementation and support across departments

All lines of enquiry were used to assess the level and consistency of implementation and support for the PSAB across departments.

PSAB program staff, asked whether managers and contracting officers understand, support, and make use of the PSAB and whether the PSAB has been implemented consistently across departments, generally indicated that most managers and contracting officers understand and support the PSAB. Some do not, however, with turnover presenting a challenge in this regard. There was disagreement as to whether the PSAB had been implemented consistently across departments, with some indicating implementation had been consistent and others that it had been somewhat inconsistent across departments.

Senior procurement officers, asked whether managers and contracting officers within their department understand, support, and make use of the PSAB, answered in the affirmative for the most part. However, some noted that not everyone is aware of the PSAB targets and that some are reluctant to contract with Aboriginal firms as they believe it does not always result in best value. Others noted the need for more support for the PSAB at the managerial level and a couple indicated that while people adhere to the requirements of the PSAB, they do not necessarily see the PSAB as being of much importance.

Very few (7%) of the PSAB coordinators who responded to the survey indicated they were aware of any instances in which their departments had failed to comply with the PSAB by not setting aside a contract valued in excess of $5,000 for which Aboriginals were the primary recipient or where their department had failed to meet their annual target. Similarly, few (11%) indicated that their department had failed to achieve their annual performance objectives in the last three years.

PSAB coordinators reported that their departments had undertaken a variety of activities as a result of the PSAB. Most commonly, these activities involved systems to monitor Aboriginal procurement, training for contracting officers, and review of procurement and contracting policies (see Table 4).

The majority (71%) of PSAB coordinators thought that all or most contracting officers within their department are familiar with the PSAB, whereas 11% thought about half were, 7% thought some were, and 11% did not know. Coordinators were also asked to characterize the support of contracting officers within their department toward various parts of the PSAB (see Table 5). Most PSAB coordinators indicated that contracting officials supported PSAB as a tool to promote Aboriginal businesses and the monitoring of Aboriginal contracts, but were neutral with respect to using the set-asides and being proactive in identifying Aboriginal suppliers.

Both the survey of PSAB coordinators and the survey of material managers indicate there are discrepancies across departments in terms of the amount of time devoted to the PSAB, the use of PSAB set-asides, and the perceived level of awareness of the PSAB required among material managers. The reported number of hours spent by PSAB coordinators on PSAB-related activities in September 2006 varied, though over half (54%) reported they had spent 5 hours or less, including 32% who spent one hour or less. Half of material managers (49%) indicated they had not used the set-aside element of the PSAB within the last year and 1 in 5 (22%) did not know or did not answer, leaving less than 1 in 3 (29%) who reported having used the set-aside at least once in the last year. The largest proportion of material managers (43%) indicated their day-to-day job requires general awareness of the PSAB, while fewer (29%) indicated a moderate level of familiarity is required. Very few (about 10%) said a high level of familiarity was required, and a similar proportion said no familiarity whatsoever was required.

The annual departmental performance reports for a select number of government departments of various sizes (n=9) were reviewed. For some of these departments, reports or data for certain years was missing or unclear, and there were some inconsistencies in terms of the nature and form in which the data is reported. Nevertheless, based on the data available, it appears that compliance with the Aboriginal procurement performance targets under the PSAB has generally improved over the years. Based on the available data, five of nine departments failed to achieve their performance targets for either 1998 or 1999 (or both years), whereas only two departments failed to achieve their performance targets in either 2004 or 2005, while other departments achieved more than double their targets in these years.

The 2004 PSAB performance report (the most recent year for which data was available) indicates that the vast majority of large departments [Note 21] are meeting or exceeding their departmental performance objectives (according to the TBS report for 2004, which excludes credit card purchases, standing offer call-ups, and amendments). The only exception is PWGSC, whose 2004 objective was $11,400,000, and which achieved $4,461,000 or 39% of this target. The summed 2004 objective for all other departments was $33,696,865, and the TBS report indicates that $24,027,000 or 71% of the objective was achieved, though in 2002 and 2003, other government departments far exceeded objectives. Nevertheless, overall, business volume directed by government departments and agencies as per the TBS report was more than double the objective. [Note 22]

3.2.4 Clarity of PSAB guidelines and criteria

Within the context of PSAB, the term "Aboriginal" includes Status and Non-Status Indians, Métis, and Inuit who are Canadian citizens and normally reside in Canada. [Note 23] The PSAB eligibility requirements outlined in a Treasury Board Contracting Policy notice in 1996 indicated that businesses could be a sole proprietorship, limited company, co-operative, partnership, or a not-for-profit organization. Regardless, they were required to meet the following criteria:

• At least 51% of the firm must be Aboriginal-owned and controlled.
• If the business has six or more full-time staff, at least one-third must be Aboriginal people.
• For joint ventures/consortiums, at least 51% of the joint venture/consortium must be owned and controlled by an Aboriginal business or businesses, as defined above. If there are six or more employees working for the joint venture/consortium, at least one-third must be Aboriginal people.
• If an Aboriginal supplier or the joint venture is to subcontract part of the requirement, the Aboriginal component of the work must be maintained. Contractors must certify that an Aboriginal firm will conduct at least one-third of the value of the work performed under the contract. [Note 24]

Following the evaluation completed in 2002, work was undertaken to further clarify the PSAB policy. In consultation with CAC, Aboriginal businesses, and an Interdepartmental Working Group, work was undertaken to clarify the eligibility requirements. A Web Notice was posted on the PSAB website summarizing clarifications that had been made to the policy and indicating that the new policy document amalgamated four previously issued Treasury Board Contracting Policy Notices (CPNs). These clarifications surrounding joint ventures and Aboriginal business identity, are summarized in Table 6.

The interviews with PSAB program staff and senior procurement officers, as well as the surveys of PSAB coordinators, material managers, and Aboriginal businesses, were used to assess the clarity of the eligibility criteria. To assess the appropriateness of the eligibility criteria, PSAB coordinators were asked to indicate whether any of the eligibility criteria hinder the Strategy's effectiveness.

PSAB program staff and senior procurement officers were asked whether the clarified PSAB policy and its guidelines and eligibility criteria are clear and consistent, whether ambiguities remain, and if so, what has been done to address these ambiguities. Program staff indicated that the clarified policy and criteria are clearer. But they also indicated there is still a need to clarify the 33% Aboriginal content requirement, noting that the requirement is that 33% of the value of the work should be performed directly by Aboriginal people, whereas some suppliers believe that they are not required to have a certain proportion of the work performed directly by Aboriginal people, provided they meet the Aboriginal ownership and control requirements. Program staff noted that a working group had been formed to address issues such as these and that there are plans to clarify this requirement and to further revise the policy. Any such clarifications will require a TB Submission.

Several senior procurement officers indicated that they were unaware of the clarifications to the policy and guidelines or that they had not reviewed them in detail. It is possible that this is in part due to the fact that the clarified policy, which consolidates four Treasury Board CPNs, is available through the PSAB website, but not found under the Contracting Policy notices and circulars on the TBS website. [Note 25] Some indicated that the policy and guidelines are now clearer and did not identify any confusion or ambiguity surrounding the eligibility requirements. Others indicated a need to improve the auditing process, which a couple of contacts indicated to be unclear, in order to ensure firms are genuinely Aboriginal.

PSAB coordinators, material managers, and Aboriginal businesses were asked about the clarity of the PSAB's eligibility criteria. Approximately one third of both PSAB coordinators and businesses indicated that the eligibility criteria were not clear (see Table 7). The vast majority (88%) of PSAB coordinators who indicated the eligibility criteria were unclear indicated that the content requirement for the labour element was unclear. Only 12% of material managers said the eligibility criteria were not clear, though 44% did not know or did not respond, suggesting they are not highly familiar with the eligibility criteria, this is not surprising, given that, as previously noted, most had not made use of the set-asides within the past year. Of the few material managers that indicated that the eligibility criteria were unclear, 65% indicated that the requirement for Aboriginal ownership and control was unclear and 53% indicated that the content requirement for the labour element was unclear.

PSAB coordinators and material managers were also asked about the clarity of the revised PSAB guidelines and policy. The largest proportion of material managers (50%) did not know whether the PSAB policy and guidelines were clear. This again suggests that many material managers are not highly familiar with the PSAB guidelines and policy or may not have been aware that the policy was recently clarified. As previously noted, many material managers indicated their day-to-day job requires only a general awareness of the PSAB. The few material managers that indicated the policy and guidelines were unclear most commonly indicated that more details were required regarding joint ventures. Most PSAB coordinators (54%) indicated the policy and guidelines were clear, while 1 in 4 (25%) indicated they were not clear (see Table 7).

PSAB coordinators were questioned whether any of the PSAB's eligibility criteria hinder the Strategy's effectiveness in increasing Aboriginal business participation in the awarding of contracts by federal departments and agencies, in response, 1 in 5 (21%) said yes (50% said no and 29% did not know). Those who indicated that the PSAB's eligibility criteria hindered its effectiveness (n=6) were asked what changes should be made to the criteria to enhance the Strategy's effectiveness. The most common suggestion, made by 5 of these 6 respondents, was to change the content requirement for the labour element to be carried out by (an) Aboriginal firm(s), which is currently 33%. The questionnaire did not ask these respondents to indicate what they thought the content requirement should be changed to, as this question is more appropriately explored in a forum that allows for discussion and for the provision of contextual qualitative information, than through a quantitative method such as a survey questionnaire.

In summary, the findings indicate that, in spite of the recent clarifications to the policy, ambiguities remain, particularly surrounding Aboriginal content requirements, and there is a continuing need to clarify the PSAB policy and eligibility requirements.

3.2.5 Mechanisms to ensure businesses meet eligibility requirements

All lines of enquiry were used to identify the mechanisms in place to ensure that businesses meet the PSAB eligibility requirements and to assess the effectiveness of these mechanisms.

PSAB program staff and senior procurement officers were asked what mechanisms are in place to ensure businesses meet the eligibility requirements, whether these mechanisms are effective, and how they could be improved. They noted that pre-award and post-award audits are conducted by Consulting and Audit Canada (CAC), with mandatory pre-award audits for contracts over $2 million and random audits conducted for those under $2 million. INAC tracks audits conducted to avoid auditing the same firms repeatedly. In cases in which no audits are conducted, the PSAB relies upon self-certification of eligibility. According to program staff, if ineligibility is revealed during a pre-award audit, the contract is generally not awarded to that particular firm, but if ineligibility is revealed during a post-award audit, there is often little to be done if the work has been completed. This is confirmed in "PSAB – Guidelines for Buyers/Government Officials", which indicates with respect to pre-award audits: "If an audit indicates that the best-assessed bidder is not in compliance with the 'Requirements for the Set-Aside Program for Aboriginal Business' the contracting authority will consider the bid non-compliant. The next best-assessed bidder would be awarded the contract if the bid meets the required conditions." [Note 26] If ineligibility is revealed during a post-award audit, however, "it is the responsibility of the contracting authority to apply any disciplinary measures as allowed in the TBS Contracting Policy". [Note 27]

A couple of program staff indicated circumstances might change between the pre-award audit and the start of the contract, raising concerns that pre-award audits are not always effective. Program staff generally indicated a need to improve these mechanisms, noting it would be useful if INAC had possession and control of a database of pre-qualified Aboriginal suppliers.

Senior procurement officers indicated that pre-award and post-award audits are conducted to determine eligibility, with many reporting they assumed these mechanisms were effective, but were unsure since they were not directly involved in the auditing process. A few senior procurement officers specified firms are also required to certify that they are Aboriginal. One contact stated that pre-award audits are more effective than post-award audits, while, in contrast, another senior procurement officer believed that large firms should be repeatedly audited post-award on a contract-by-contract basis. The "Guidelines for Buyers/Government Officials" indicates that INAC is ultimately responsible for verifying the eligibility of bidders, but notes that the requisitioning office or contracting authority may request verification if they wish to do so; however, some respondents indicated they never requested audits because they were unclear as to the process and it seemed to be a lot of work. Similarly to PSAB program staff, some senior procurement officers indicated there should be a list of pre-qualified Aboriginal suppliers, with a few unsure whether or not such a list already existed.

INAC provided data on the eligibility audits conducted for the PSAB between 1997 and 2005, which are presented in Table 8. In most years, more pre-award than post-award audits were conducted. The total percentage of non-compliant audits ranged from 8% to 33% in these years. No data was made available regarding how non-compliant audits were handled, since, as previously noted, these decisions ultimately rest with the purchasing department. Similarly, no data was available to determine whether or not there were firms that repeatedly or habitually failed audits who continued to be awarded federal contracts under the PSAB – if this were occurring, it would, of course, be a threat to the credibility and effectiveness of the PSAB in enhancing Aboriginal business development. The evaluation completed in 2002 recommended that the monitoring and enforcement of Aboriginal content criteria, in particular, be improved; as such, work was undertaken to clarify the Aboriginal content requirements. However, it does not appear that this work further elaborated sanctions for non-compliance. As such, there remains the potential to increase transparency surrounding the audit process and to institutionalize consequences for non-compliance, which might reassure Aboriginal businesses that the set-aside award process is fair and that there are appropriate measures being taken against those who attempt abuse under the PSAB.

Nearly half (44%) of businesses surveyed indicated that their firm had undergone at least one pre- or post-award audit of their eligibility for contracts under the PSAB. Of these, most reported they had been audited once (55%) or twice (18%).

3.2.6 Challenges in supporting and implementing the PSAB

Both the interviews with the senior procurement officers as well as the surveys of PSAB coordinators and material managers were used to identify challenges faced by departments in supporting and implementing the PSAB.

Senior procurement officers, asked what challenges contracting managers in their departments face in supporting the PSAB, noted lack of buy-in and awareness from some line managers and procurement officials. They indicated that the objectives of the PSAB sometimes seem contradictory with initiatives focused on saving money, such as The Way Forward, resulting in the belief among some managers that setting aside contracts under the PSAB is not conducive to achieving best value. A couple noted issues surrounding lack of Aboriginal capacity in certain fields, with one senior procurement officer noting it is difficult to assure best value when there is only one Aboriginal firm with the capacity to complete a particular contract and others noting it can be challenging and time-consuming to ascertain whether there is Aboriginal capacity in particular areas. There is sometimes reluctance to contract Aboriginal firms for services in particular. One senior procurement officer remarked that unclear audit recommendations do not provide definitive answers as to eligibility, thus presenting a challenge. Another commented that certain purchasing and reporting processes and conventions, such as attributing purchases to the section that does the purchasing rather than the section for which the purchase is intended, and the fact that PWGSC makes purchases on behalf of government departments in certain domains, results in confusion in some cases as to who is ultimately responsible for Aboriginal procurement.

As shown in Figure 2, for the most part, PSAB coordinators and material managers did not indicate challenges in adhering to the PSAB. For the most part, results from the groups were consistent, given that the three issues that were most commonly identified as challenges were the same: upholding competition among firms; promoting best value for money; and, encouraging more firms to bid on work. However, a greater proportion of material managers indicated that adhering to the PSAB had created challenges with respect to encouraging more firms to bid on work (29% of material managers compared to 14% of PSAB coordinators).

Coordinators and Managers: Challenges in adhering to PSAB

Text desctiption - Coordinators and Managers: Challenges in adhering to PSAB

This figure shows a graph comparing PSAB Coordinators and Material Managers identified challenges in adhering to PSAB.

The following 7 statements are along the vertical y axis:

• "upholding competition among firms"
• "promoting best value for money"
• "encouraging more firms to bid on work"
• "maintaining openness and transparency in the bidding process"
• "executing contracts quickly"
• "ensuring quality of goods and services"
• "meeting other obligations required for federal procurement"

Along the x axis are the percentage rates, in 10% increments, for "yes" responses. Bars representing PSAB coordinator responses are next to the first six statements, with bars representing the Material Manager responses underneath and included next to all seven statements. 

The statement "upholding competition among firms," was identified as a challenge by 29% of the 28 PSAB coordinators who participated and 22% of the 144 Material Managers who participated. For the statement "promoting best value for money," 21% of the 28 PSAB coordinators who participated and 26% of the 144 Material Managers who participated indicated "yes" this was a challenge.

Concerning "encouraging more firms to bid on work," 14% of the 28 PSAB coordinators who participated and 29% of the 144 Material Managers who participated indicated that this was a challenge.

"Maintaining openness and transparency in the bidding process" was identified as a challenge by 11% of the 28 PSAB coordinators who participated and 13% of the 144 Material Managers who participated.

For the statement "executing contracts quickly," 11% of the 28 PSAB coordinators who participated and 16% of the 144 Material Managers who participated indicated "yes" this was a challenge.

7% of the 28 PSAB coordinators who participated and 15% of the 144 Material Managers who participated indicated that "ensuring quality of goods and services" was a challenge.

Finally, for the statement "meeting other obligations required for federal procurement," 19% of the 144 Material Managers who participated indicated "yes" it was a challenge. This was the only statement where the responses of PSAB coordinators were not displayed.

About half of PSAB coordinators indicated that a lack of contracts suitable for Aboriginal suppliers and a lack of resources to promote opportunities to Aboriginal businesses constituted barriers in implementing the PSAB, though less than one fifth (18%) each indicated these were major barriers. Most (50%) indicated the need to meet other contracting objectives did not constitute a barrier; of the one third (32%) that indicated this was a barrier, most indicated it constituted a minor barrier.

As previously noted, senior procurement officers noted that lack of buy-in and awareness from some line managers and procurement officials were challenges. The low response rate to the survey among material managers in spite of two reminders and the provision of two options for completing the survey (electronically or on paper) similarly suggest that discontent, or at best, indifference or lack of awareness of the PSAB on the part of a number of material managers may have also been a significant challenge in supporting and implementing the PSAB. The evaluation completed in 2002 experienced similar challenges with respect to low response rates, and ultimately recommended that a renewed Government of Canada commitment to the strategy be obtained via a presentation to the Deputy Ministers' Committee on Aboriginal Policy (DMCAP) or another suitable forum; however, this activity was postponed and does not seem to have taken place, and the introduction of performance pay penalties, which was also explored as an option for enhancing senior officials' commitment to fostering the use of the PSAB among their departmental procurement personnel, was ultimately rejected.

3.3 Impact/Success

The following section addresses PSAB's impact and success, including the extent to which it has achieved its expected outcomes, unintended outcomes that have been observed, and differences in impact across sectors and locations.

3.3.1 Role in increasing Aboriginal business participation in federal contracting and business growth and development

The interviews with PSAB program staff and senior procurement officers, as well as the survey of Aboriginal businesses were used to gain insight into how PSAB had increased Aboriginal business participation in the awarding of contracts by federal departments and the extent to which PSAB had contributed to Aboriginal business growth and development.

PSAB program staff, asked how PSAB had increased Aboriginal business participation in the awarding of contracts by federal departments and agencies, and to what extent PSAB has contributed to Aboriginal business growth and development, indicated that the PSAB had definitely increased the number of Aboriginal businesses doing business with the federal government. They noted that the number of Aboriginal businesses in Canada had grown tenfold since the implementation of the PSAB in 1996, with Aboriginal businesses now represented to some extent in virtually all sectors of the economy. PSAB program staff believe that some companies established around the same time as the PSAB might not have been successful had it not been for the Strategy. Some noted that the fact that there are now fewer set-aside contracts indicated that Aboriginal firms have become more successful in the open market and that anecdotal evidence and "success stories" support the notion that PSAB greatly contributed to Aboriginal growth and development. Program staff acknowledged, however, hard data available to support these claims is lacking, and indicated more needs to be done to measure the PSAB's impacts.

Senior procurement officers, asked how the PSAB had increased Aboriginal business participation within their departments, mostly indicated this had been accomplished through the setting of targets, and though the majority indicated targets had been met primarily or exclusively through incidental procurement rather than through set-asides, some indicated that their department had promoted and made use of set-asides. Some noted they had seen a dramatic increase in the number of Aboriginal firms participating in government procurement and winning contracts under open competition, though one contact noted that commitment to the PSAB had waned recently due to a need to focus on other priorities.

Asked to what extent the PSAB had enhanced the profitability of their business, over one third (36%) of Aboriginal business survey respondents indicated to some extent, whereas one third (32%) indicated it had not at all enhanced profitability, and one quarter (24%) indicated it had enhanced the profitability of their business to a great extent.

3.3.2 Achievement of expected direct and indirect impacts

The document review and the database assessment were used to assess the achievement of the PSAB's expected direct and indirect impacts; however, little supporting data was found to be available in this regard. As such, the survey of Aboriginal businesses that received contracts under the PSAB constitutes an important data source as to the impact of the PSAB. In addition, PSAB program staff, senior procurement officers, and PSAB coordinators were asked to provide and support their assessments of the extent to which the PSAB had achieved its expected direct and indirect impacts.

Though data supporting the achievement of most expected impacts is lacking, the increase in the number and value of federal contracts awarded to Aboriginal firms following the inception of the PSAB is well documented. As shown in Table 9, though the number of contracts and the dollar value of the contracts have fluctuated in recent years, they were significantly higher in 2004 than in 1997.

Table 10 shows that contracts with Aboriginal firms awarded by PWGSC have increased from $32 million in 1996 to $401 million in 2006.  The share of PSAB in awards to Aboriginal firms has fluctuated from a high of 72% in 1998 to 6% in 2006 (discounting the start-up year).

Table 10 shows that the PSAB has a declining proportion of total Aboriginal contracting, having peaked in 1998, and suggests that the set-aside contracts under the Strategy have become less important in supporting the activity of Aboriginal businesses in federal contracting. Unfortunately, only data on the contracting activity of PWGSC was available for analysis.

Table 11 shows the number and percentage share of contracts that were awarded to Aboriginal compared to non-Aboriginal firms of all the contracts awarded by PWGSC between 1996 and 2006. [Note 28] The table indicates that the proportion of contracts awarded to Aboriginal firms was negligible (less than 1%) in 1996, but increased to 4% by 2003 and has remained at this proportion since then. Similarly, the proportion of contracting dollars to Aboriginal firms, which was negligible in 1996 and 1997, has increased to 5% in 2006, though the proportion has fluctuated somewhat in recent years.

The most direct assessment of the impacts of the PSAB is provided through the survey of businesses that received contracts under the Strategy. Only one-third (33%) of business respondents indicated that their business had experience contracting with the federal government prior to the start of the PSAB in April 1996, though nearly one-third (30%) indicated this was not applicable since their business had not been in existence that long.

Businesses were asked to indicate how many contracts their firm had been awarded as set-asides, under open competition, and in total within the past two years. The results, presented in Table 12, included a significant proportion that did not know or did not respond to these questions.

Businesses were also asked to indicate what percentage of their revenue within the last two years had been due to the PSAB set-asides. Just over one quarter (26%) indicated that 0% of their revenue was due to PSAB set-asides, while nearly one tenth (9%) indicated that 100% of their revenue had been due to PSAB set-asides and one seventh (15%) did not respond. The median was 13% and the mean was 31%. This suggests that there are, as speculated, some firms that have become highly reliant upon the PSAB. But it is more common for firms that received at least one contract under the PSAB since the Strategy's implementation to have no reliance whatsoever upon the PSAB for revenue. Businesses were asked about the nature of their contracts and revenue for the last two years only to minimize the potential for inaccurate data due to respondents being unable to recall this information accurately. However, many of the respondents did not provide this data. In order to assess the long-term impact of an intervention, a longitudinal study, with repeated testing, is most effective and informative. The data collected surrounding PSAB-related revenue and contracts provides a snapshot of the impact in this regard over the past couple of years, but does little to assess the long-term impact.

To help assess the more long-term impact the PSAB has had upon businesses that received at least one contract under the PSAB since the implementation of the Strategy, business respondents were asked to indicate the overall impact the PSAB had had on a number of aspects of their businesses. This constitutes self-reported data directly from businesses as to the extent to which the PSAB is achieving its expected impacts. As shown in Figure 3, the three most positive reported impacts of the PSAB were firms' ability to compete for federal government contracts and the number and value of federal contracts awarded to the firms, with over half of respondents indicating that the PSAB had had very or somewhat positive impacts in these areas. Close to half also indicated that the PSAB had positively impacted job creation within their firm and their firm's capacity to undertake larger contracts. About 4 in 10 indicated PSAB had had a positive impact on employment stability within their firms, their firms' ability to win contracts at other levels of government, and the number and quality of partnerships between their firms and non-Aboriginal businesses.

Most positive reported impacts of PSAB for firms (Businesses, n=92)

Text description - Most positive reported impacts of PSAB for firms (Businesses, n=92)

This figure shows a graph of the most positive reported impacts of PSAB for firms.

The following nine statements appear along the y axis:

• "your firm's ability to compete for federal government contracts"
• "number of federal contracts awarded to your firm"
• "value of federal contracts awarded to your firm"
• "job creation within your firm"
• "your firm's capacity to undertake larger contracts"
• "employment stability within your firm"
• "your firm's ability to acquire contracts at other levels of government"
• "the quality of partnerships between your firm and non-Aboriginal businesses"
• "number of partnerships between your firm and non-Aboriginal business"

There were 92 businesses who responded to these statements. Along the x axis are their response rates in 10% increments.

For the statement "your firm's ability to compete for federal government contracts,"
40% of respondents reported a very positive impact in this area, 22% of respondents reported a somewhat positive impact in this area and 21% of respondents reported no impact in this area.

For the statement "number of federal contracts awarded to your firm," 33% of respondents reported a very positive impact, 25% of respondents reported a somewhat positive impact and 17% of respondents reported no impact.

Concerning the statement "value of federal contracts awarded to your firm," 28% of respondents reported a very positive impact in this area, 29% of respondents reported a somewhat positive impact in this area and 20% of respondents reported no impact in this area.

For the statement "job creation within your firm," 28% of respondents reported a very positive impact in this area, 19% of respondents reported a somewhat positive impact in this area and 32% of respondents reported no impact in this area.

Regarding the impact statement "your firm's capacity to undertake larger contracts," 21% of respondents reported a very positive impact in this area, 26% of respondents reported a somewhat positive impact in this area and 28% of respondents reported no impact in this area.

For the statement "employment stability within your firm," 23% of respondents reported this as a very positive impact, 19% of respondents reported this as a somewhat positive impact and 36% of respondents reported no impact.

Concerning the statement "your firm's ability to acquire contracts at other levels of government," 11% of respondents reported a very positive impact in this area, 29% of respondents reported a somewhat positive impact in this area and 37% of respondents reported no impact in this area.

For the statement "the quality of partnerships between your firm and non-Aboriginal businesses," 19% of respondents reported a very positive impact in this area, 20% of respondents reported a somewhat positive impact in this area and 37% of respondents reported no impact in this area.

Finally for the statement "number of partnerships between your firm and non-Aboriginal business," 17% of respondents reported a very positive impact in this area, 22% of respondents reported a somewhat positive impact in this area and 36% of respondents reported no impact in this area.

Figure 4 shows areas in which firms were less likely to report that the PSAB had had a positive impact. Only about one third of respondents indicated that PSAB had positively affected investment in their firms, their firms' ability to win non-government contracts, their firms' ability to acquire loans and other financing, and the amount of administrative work required to undertake government contracts. Over one fifth (22%) indicated the PSAB had had a negative impact on the amount of administrative work required, with some specifying that the paperwork and administrative requirements for the PSAB were time intensive. The least reported positive impact of the PSAB (23%) was the ability of firms to develop product lines to meet new demands.

Least positive reported impacts of the PSAB for firms (Businesses, n=92)

Text description - Least positive reported impacts of the PSAB for firms (Businesses, n=92)

This figure shows a graph of the areas in which firms were less likely to report that PSAB had a positive impact.

The following five statements appear along the y axis:

• "investment in your firm"
• "your firm's ability to win non-government contracts"
• "the amount of administrative work required to undertake government contracts"
• "your firm's ability to acquire loans and other financing"
• "the ability of your firm to develop product lines to meet new demands"

There were 92 businesses who responded to these statements. Along the x axis are their response rates in 10% increments.

For the impact statement "investment in your firm," 15% of respondents reported a very positive impact in this area, 17% of respondents reported a somewhat positive impact in this area and 36% of respondents reported no impact in this area.

For the statement "your firm's ability to win non-government contracts," 7% of respondents reported a very positive impact in this area, 25% of respondents reported a somewhat positive impact in this area and 42% of respondents reported no impact in this areat.

Concerning the statement "the amount of administrative work required to undertake government contracts," 14% of respondents reported a very positive impact in this area, 16% of respondents reported a somewhat positive impact in this area and 30% of respondents reported no impact in this area.

For the statement "your firm's ability to acquire loans and other financing," 12% of respondents reported a very positive impact in this area, 16% of respondents reported a somewhat positive impact in this area and 47% of respondents reported no impact in this area.

Finally for the statement "the ability of your firm to develop product lines to meet new demands," 5% of respondents reported a very positive impact in this area, 17% of respondents reported a somewhat positive impact in this area and 48% of respondents reported no impact in this area.

Return to Figure 4

The results of the business survey indicate that over half of respondents thought that PSAB had positively affected their ability to compete for federal government contracts, as well as the number and value of contracts awarded to their firms. Still, approximately 4 in 10 respondents did not indicate that they had benefited from the PSAB in these ways, and it is possible, the survey respondents may not be reflective of the sample as a whole, as it is quite possible that those who reaped the most benefits from the PSAB would have been more inclined to respond to the questionnaire than those who did not think they had benefited from the Strategy. Furthermore, these three areas were the only ones that over half of survey respondents indicated as having been positively affected by the PSAB. For all other impacts, less than half of survey respondents reported that the PSAB had a positive impact, and it appears that, for the most part, the expected indirect impacts of the PSAB have not come to fruition, given that only about one-third or less of business respondents reported that the PSAB had positively affected things like investment in their firms, their firms' ability to win non-government contracts, and the ability of their firms to develop product lines to meet new demands.

Business respondents were asked to indicate the number of full and part-time jobs that had been created for Aboriginal and non-Aboriginal people as a result of contracts awarded to their firm under the PSAB. As shown in Table 13, although sizeable proportions of respondents did not provide this information or indicated that no jobs had been created, about one third of respondents indicated that full time and/or part time jobs for Aboriginals had been created as a result of the PSAB.

Business respondents were also asked their opinions as to the impact of the PSAB upon Aboriginal businesses and communities generally (see Figure 5, next page). Generally, a significant proportion of respondents (between over one quarter to nearly one half) did not know the impacts or indicated they did not believe there were any impacts (in most cases about one fifth). Approximately 4 in 10 believed that the PSAB had positively impacted awareness of Aboriginal businesses within the federal government, dependency on social assistance within the Aboriginal population, the number of profitable Aboriginal-owned businesses, the rate of employment within Aboriginal communities, and awareness among Aboriginal businesses of procurement opportunities in the federal government. They were slightly less inclined to believe that the PSAB had positively impacted the overall prosperity of Aboriginal communities, the ability of Aboriginal firms to attract and retain qualified Aboriginal employees, the pool of qualified Aboriginal candidates, and particularly, the ability of Aboriginal firms, in remote locations, to do business with the federal government.

Impacts of PSAB on Aboriginal businesses and communities (Businesses, n=92)

Text description - Impacts of PSAB on Aboriginal businesses and communities (Businesses, n=92)

This figure shows a graph of business respondents' opinions on the impact of the PSAB upon Aboriginal businesses and communities. There were 92 businesses who gave responses.

The following nine statements appear along the y axis:

• "awareness of Aboriginal businesses within the federal government"
• "dependency on social assistance within the Aboriginal population"
• "number of profitable Aboriginal-owned businesses"
• "rate of employment within Aboriginal communities"
• "awareness among Aboriginal businesses of procurement opportunities with the federal government"
• "the overall prosperity of Aboriginal communities"
• "the ability of Aboriginal firms to attract and retain qualified Aboriginal employees"
• "the pool of qualified Aboriginal candidates"
• "the ability of Aboriginal firms in remote locations to do business with the federal government"

Along the x axis are response rates in 10% increments.

For the statement "awareness of Aboriginal businesses within the federal government," 15% of respondents were of the opinion that there was a very positive impact in this area, 29% of respondents were of the opinion that there was a somewhat positive impact in this area, 20% of respondents were of the opinion that there was no impact in this area, and 28% of respondents either did not know or did not respond.

For the statement "dependency on social assistance within the Aboriginal population," 17% of respondents reported a very positive impact in this area, 25% of respondents reported a somewhat positive impact in this area, 24% of respondents reported no impact in this area, and 29% of respondents either did not know or did not respond.

Concerning the statement "number of profitable Aboriginal-owned businesses," 13% of respondents reported a very positive impact in this area, 28% of respondents reported a somewhat positive impact in this area, 17% of respondents reported no impact in this area, and 36% of respondents either did not know or did not respond.

For the statement "rate of employment within Aboriginal communities," 13% of respondents were of the opinion that there was a very positive impact in this area, 27% of respondents were of the opinion that there was a somewhat positive impact in this area and 20% of respondents were of the opinion that there was a no impact in this area, and 36% of respondents either did not know or did not respond.

Regarding the statement "awareness among Aboriginal businesses of procurement opportunities with the federal government," 12% of respondents reported a very positive impact in this area, 28% of respondents reported a somewhat positive impact in this area and 17% of respondents reported no impact in this area, and 35% of respondents either did not know or did not respond.

Concerning the statement "the overall prosperity of Aboriginal communities," 11% of respondents reported a very positive impact in this area, 25% of respondents reported a somewhat positive impact in this area, 24% of respondents reported no impact in this area, and 36% of respondents either did not know or did not respond.

For the statement "the ability of Aboriginal firms to attract and retain qualified Aboriginal employees," 11% of respondents reported a very positive impact in this area, 24% of respondents reported a somewhat positive impact in this area and 23% of respondents reported no impact in this area, and 35% of respondents either did not know or did not respond.

For impacts regarding "the pool of qualified Aboriginal candidates," 9% of respondents were of the opinion that there was a very positive impact in this area, 26% of respondents were of the opinion that there was a somewhat positive impact in this area and 23% of respondents were of the opinion that there was no impact in this area, and 32% of respondents either did not know or did not respond.

Finally for the statement "the ability of Aboriginal firms in remote locations to do business with the federal government," 7% of respondents reported a very positive impact in this area, 22% of respondents reported a somewhat positive impact in this area and 20% of respondents reported no impact in this area, and 47% of respondents either did not know or did not respond.

PSAB program staff and senior procurement officers were asked to what extent PSAB has achieved its expected direct and indirect impacts. Program staff noted that while there is data to support a few of the expected direct impacts, such as increases in the numbers and values of contracts awarded to Aboriginal businesses and in the number of Aboriginal firms competing for and winning contracts, there are large discrepancies between the expected impacts and the current performance data being collected. While anecdotal evidence (e.g. the profiles of successful Aboriginal businesses presented in PSAB performance reports) appears to indicate that capacity has improved and that more partnerships are being formed, clear data is lacking. Most program staff indicated that there was no data available regarding the expected indirect benefits. Furthermore, they stated that counting standing offers for which there have not been call-ups can be misleading.

Senior procurement officers were asked to what extent contracting by their department had assisted the PSAB in achieving its expected direct and indirect outcomes. A couple of officers were unsure about the extent to which some expected impacts, such as increased opportunities in other markets and the development of new product lines, were being achieved. Some indicated that there had been a positive impact on the number of firms competing for and winning contracts, as well as the value of the contracts. They also believed that some of the indirect expected impacts would have followed as a result of this, although they did not have data available to confirm it. One person noted that some strong joint ventures with genuine commitments had been observed, which had led to stronger Aboriginal businesses. A couple of contacts indicated that the number of Aboriginal firms had increased, and that these firms are now better promoting themselves to the public sector, although a couple of other contacts questioned how much impact government procurement alone could have, noting that there was not very much variety in the types of goods being purchased from Aboriginal suppliers.

PSAB coordinators generally believed that the PSAB had had at least somewhat positive impacts:

• The vast majority (86%) believed PSAB had positively influenced the number of Aboriginal firms competing for contracts, most of whom indicated the PSAB had a somewhat positive impact in this regard.
• Over four-fifths (82%) believed the PSAB had positively influenced the number of contracts awarded to Aboriginal firms, including over one third who indicated PSAB had had a very positive impact; nearly the same proportion (79%) indicated the PSAB had positively influenced the value of contracts awarded to Aboriginal firms, including one quarter who indicated PSAB had had a very positive impact.
• Three-quarters believed PSAB had positively influenced the number of Aboriginal firms winning federal government contracts, including one quarter who indicated this had been a very positive impact.
• Six-tenths (61%) believed the PSAB had positively influenced the extent to which Aboriginal businesses are developing product lines to meet new demands, the vast majority of whom believed this had been a somewhat positive impact; one third (32%) did not know. The same proportion (61%) believed the PSAB had had a positive impact upon the capacity of Aboriginal firms, including one quarter who believed this had been a very positive impact.
• Over half (57%) believed the PSAB had had a positive impact upon investment in Aboriginal firms and business start-ups, while nearly one third (29%) did not know. A similar proportion (54%) indicated the PSAB had positively impacted opportunities for Aboriginal firms in other markets, while over one third (36%) did not know.

3.3.3 Unintended outcomes

Interviews with PSAB program staff and senior procurement officers were used to explore whether there had been any unintended outcomes, either positive or negative, as a result of the PSAB. In particular, the interviews were used as an opportunity to explore the extent to which key informants think that "shell companies" and illegitimate joint ventures have proliferated as a result of the PSAB.

PSAB program staff were asked whether any unintended outcomes, positive or negative, occurred as a result of the PSAB. They noted the following unintended outcomes:

• Shell companies and illegitimate joint ventures- While some did not think that shell companies posed a large problem, others noted that shell companies are not being appropriately penalized for their dishonesty.
• Reliance upon set-asides- Some program staff noted that some Aboriginal firms have come to rely upon the set-asides, which impedes their growth.
• Small Aboriginal businesses have difficulty competing- It was noted that small Aboriginal businesses sometimes have difficulty competing with larger, more established Aboriginal businesses for PSAB set-asides.
• Confusion surrounding Aboriginal content- A couple of program staff noted that the confusion surrounding how to define Aboriginal content had been a negative unintended outcome.
• Frustration with government bureaucracy- A couple of program staff noted that one negative intended outcome was frustration with government bureaucracy on the part of Aboriginal businesses.
• More positive relations between the federal government and Aboriginal businesses- On the other hand, a couple of program staff indicated that a more positive rapport between government and Aboriginal businesses had developed as a result of the PSAB.

Senior procurement officers, asked whether any unintended outcomes, positive or negative, occurred as a result of the PSAB and whether their department had requested eligibility audits for any companies receiving contracts under the PSAB (such as suspected shell companies), generally did not identify any unexpected impacts. Many indicated that shell companies had not been a problem for their department, though some noted this was in part because their department seldom if ever used set-aside contracts. Others indicated that shell companies were of some concern, but some indicated that they did not request eligibility audits because they assumed that, under standing offers, firms were pre-qualified, because they were under the impression that audits were being conducted regardless, or because, as previously indicated, they found the auditing process unclear.

3.3.4 Impacts across sectors and locations

The database assessment, as well as the survey of Aboriginal business and interviews with PSAB program staff was used to assess the differences in the impact of the PSAB across locations and industry sectors and to gain insight into whether there are certain types of Aboriginal businesses that the PSAB has failed to help.

The database assessment indicated half of all PSAB vendors (47%) were located in either Manitoba or Ontario, while one-fifth (22%) were from Quebec or Alberta. Non-PSAB vendors were somewhat more evenly distributed throughout the country. Ontario had the largest share (27%), while British Columbia, Alberta, Quebec, and Manitoba all had between 11% and 13% each of the non-PSAB vendors. Both PSAB and non-PSAB vendors fell into a variety of industry sectors. The construction industry was most prominent, with 12% of PSAB vendors and 14% of non-PSAB vendors. For 23% of PSAB vendors, the Industry Sector Description was classified as "Unknown."

The vast majority (79%) of business survey respondents indicated that they are located within 100 km of an urban centre of 5,000 people or more. Nearly half (46%) indicated they are located in an urban centre, while just over one-third (34%) indicated they were located on reserves. Similarly to the database assessment, the construction industry was the most prominently represented sector (28%), followed by manufacturing (20%). [Note 29]

PSAB program staff were asked whether they believe differences in impacts exist across sectors and locations, and whether there are certain types of Aboriginal businesses PSAB has failed to help. Some were unsure, noting a lack of data, whereas a few noted that urban firms, and particularly those in the National Capital Region, tend to benefit the most, but indicated that this is to be expected given most sourcing is out of Ottawa.A couple indicated that demand for IT products and services was high, but generally the PSAB was seen as being less successful in the medical field, as well as in the architectural, engineering, and surveying fields. PSAB program staff see a continuing need to build Aboriginal capacity in general, as well as in the skilled trades. It was noted that the government tends to do construction through large, established non-Aboriginal companies, many of whom are reluctant to hire Aboriginal employees due to negative stereotyping. As such, it was reported the PSAB has done little to help Aboriginal plumbers, carpenters, electricians, and cooling and refrigeration technicians.

3.3.5 Trends in number and dollar value of contracts awarded to Aboriginal businesses

As shown in Table 14, in recent years, the dollar value, and more notably, the number of contracts awarded to Aboriginal-owned firms has declined. PSAB program staff were asked what accounts for the recent decline in the number and dollar value of contracts awarded to Aboriginal businesses. Some indicated these trends in part reflect a decline in federal spending and a shift from awarding many small contracts to awarding a smaller number of large contracts through standing offers. Some also indicated that the number of firms winning contracts under open competition has increased, and a couple noted that the data can be misleading because it sometimes excludes credit card purchases and purchases made directly by departments. A couple of program staff, however, attributed the recent decline in the number and dollar value of contracts awarded to Aboriginal businesses to a diminishing commitment to PSAB and to Aboriginal procurement on the part of government departments, as well as to turnover in the management of the Strategy that had occurred.

The PWGSC data presented in Section 3.3.2 (Table 11) confirms that there was a significant drop in the dollar value of Aboriginal procurement from 2003 to 2004, but indicates that there was a recovery in 2005 and 2006.

The nature of Aboriginal procurement has changed over the years. The document review indicates that, as shown in Table 15, though the number of set-aside contracts valued at greater than $25,000 has declined after peaking in 2001, the number of incidental contracts (non-set aside contracts awarded to Aboriginal businesses) was at an all time high in 2004 (the most recent year for which data was available). The data in Table 15, as well as the PWGSC data presented in Table 10, indicate that set-aside contracts under the PSAB represent a decreasing proportion of Aboriginal contracting, whereas incidental procurement to Aboriginal firms has become increasingly prominent.

3.4 Cost-effectiveness/lessons learned/alternatives

The following section presents key informant perceptions of the cost-effectiveness of the PSAB, as well as lessons learned to date and alternative means of achieving the PSAB's objective.

3.4.1 Costs of delivering and complying with the PSAB

Given that the PSAB does not have dollars directly attached to it, interviews with PSAB program staff and senior procurement officers were used to ascertain the types of costs involved in delivering and complying with the PSAB.

PSAB program staff, asked what the costs, human and financial, are of delivering the PSAB, generally indicated that overall, the cost is small. Costs include human resources (10 people across the regions and 10 people in headquarters in Ottawa), putting together publications and having material translated, and an MOU with Consulting and Audit Canada (CAC) under which the eligibility audits, of a minimum cost of $100,000 per year, are conducted. Other costs include the cost of bringing together and accommodating PSAB coordinators for conferences, and the transportation and accommodation costs involved when PSAB staff travel to deliver presentations. Program staff noted there are no dollars attached to PSAB, in that no funding from Treasury Board is provided for the Strategy, but rather, the PSAB is now funded with internal INAC resources.

Senior procurement officers generally indicated the costs of complying with the PSAB are minimal, though some noted costs related to time spent educating and providing guidance to procurement officers, preparing statistics and reports, attending CAC debriefings, and undertaking promotional activities. Senior procurement officers also indicated that sometimes, when a set-aside competition fails to yield compliant bids and the contract must be re-tendered, two RFP processes must be undertaken, which involves extra time and costs. A couple of contacts indicated that prices are sometimes higher (one person estimated them to be 5-10% higher) under PSAB set-aside contracts than they would be under competition.

3.4.2 Consistency with Treasury Board Contracting Policy

Interviews with PSAB program staff and senior procurement officers were used to ascertain what inconsistencies, if any, there are between the PSAB and Treasury Board Contracting Policy.

PSAB program staff and senior procurement officers were asked whether there were any challenges in ensuring the PSAB is consistent with Treasury Board (TB) Contracting Policy. A couple of program staff did not perceive any challenges, while others noted confusion regarding the definition of Aboriginal content and tension between the PSAB and goals surrounding value for money embodied, in particular, by the PWGSC initiative The Way Forward. Under The Way Forward, 10 categories of goods and services had to be purchased by PWGSC under mandatory standing offers and the threshold at which the Procurement Review Committee could review contracts was raised from $2 million to $5 million, reducing opportunities to advocate on behalf of Aboriginal businesses. According to program staff, the standing offer criteria under The Way Forward, which were incorporated into TB Contracting Policy, focused upon volume-based price reductions and widespread business presence, and were detrimental to Aboriginal businesses and small businesses in general. In fact, due to the negative response from small and medium-sized businesses, alternative cost-saving strategies in lieu of The Way Forward are now being considered. It was noted that PSAB is expected to align with TB policy, and that substantive changes to the Strategy can only be made via a Memorandum to Cabinet (MC).

About half of the senior procurement officers did not perceive any tensions between PSAB and TB Contracting Policy, while others indicated that there is sometimes tension between supporting Aboriginal firms and getting the best value for money, favouring the free market, giving equal opportunity to all businesses, and respecting trade agreements.  Nevertheless, a couple of senior procurement officers noted that this is an inherent tension and akin to the tensions between green procurement and these values. One person noted that, even under the PSAB, it is possible to negotiate prices down and there is no obligation to award a contract unless best value can be achieved. Finally, a couple of senior procurement officers noted ongoing confusion regarding use of Comprehensive Land Claims Agreements and their relationship to the PSAB.

3.4.3 Challenges in setting aside contracts and procuring goods and services from Aboriginal firms

PSAB coordinators and material managers were asked a series of questions to identify barriers to setting aside contracts and procuring goods and services from Aboriginal firms. To identify additional barriers and to collect additional qualitative and contextual data in this regard, senior procurement officers were also given the opportunity to identify during their interviews challenges their department faced in setting aside contracts under the PSAB.

PSAB coordinators and material managers surveyed agreed that the availability of qualified Aboriginal suppliers constitutes a barrier to procuring goods and services from Aboriginal businesses. Three quarters of PSAB coordinators and two thirds (67%) of material managers indicated that this constituted at least a minor barrier, and a significant proportion of these respondents indicated that it constituted a major barrier (see Table 16). Most PSAB coordinators and material managers indicated that upholding Treasury Board Contracting Policy, illegitimate joint ventures, and obtaining value for money were not barriers, or did not know whether or not these were barriers.

Table 16: Barriers to procuring goods and services from Aboriginal businesses
(PSAB Coordinators and Material Managers)

The evaluation completed in 2002 also revealed challenges with respect to Aboriginal capacity, yielding a recommendation that INAC create capacity development tools that would help Aboriginal firms take advantage of contracting opportunities with the federal government, including those set aside under the PSAB. In response, INAC created the Web-based Guide to Doing Business with the Federal Government, an interactive training module designed with a view to helping potential bidders understand the federal procurement process and how to market their services to government. Work has also begun on the drafting of an Aboriginal Benefits Policy, which will be elaborated upon in the following section.

Senior procurement officers, asked what challenges and barriers exist for their department in setting aside contracts under the PSAB, often did not find this question to be relevant, since many indicated that their department primarily or exclusively met their Aboriginal procurement objectives through incidental procurement, rather than through set-asides. Others reiterated the challenges noted in regard to supporting the PSAB, such as concerns with achieving best value and not knowing whether Aboriginal capacity exists in certain domains. It was also noted that it can sometimes be somewhat subjective as to whether a contract is primarily destined for an Aboriginal population and should be set aside, and disagreements with PWGSC sometimes arise as to whether to set contracts aside. In light of these challenges, senior procurement officers noted the need for PSAB to work more closely with PWGSC, the need to raise PSAB awareness among contracting officers and Responsibility Centre Managers, and for supplier inventories of qualified Aboriginal firms to be made available.

3.4.4 Alternative means of achieving the PSAB's objective

Both PSAB program staff and senior procurement officers were asked whether there are alternative means of achieving the PSAB's objective of fostering Aboriginal business development and what the implication of implementing such means would be.

Program staff indicated that an Aboriginal benefits strategy would complement the PSAB by ensuring Aboriginal people benefited in some way from large contracts. This strategy would be akin to the government's industrial and regional benefits policy, under which procurement has been used to foster industrial and regional development. Under such an Aboriginal benefits strategy, an Aboriginal benefits requirement would be included in large contracts whereby contractors would be required to demonstrate some commitment to benefiting Aboriginal people in order to be awarded these contracts, through, for example, establishment of a scholarship program, the creation of jobs for Aboriginal people, or a commitment to acquire materials and supplies from Aboriginal vendors. It is believed that such a strategy would ultimately improve existing Aboriginal business capacity. To date, this requirement was built into one contract; however, in order for an Aboriginal benefits requirement to be added under the PSAB, Cabinet approval would need to be sought via a Memorandum to Cabinet (MC).

Other alternative means of fostering Aboriginal business development mentioned by program staff include providing more grants and contributions to Aboriginal businesses and requiring businesses to meet Aboriginal staffing quotas.  Still, some emphasized that the PSAB is still needed, given the high density of Aboriginal youth who will continue to need the PSAB to help them gain experience and develop capacity.

Senior procurement officers were generally unable to identify alternative means of achieving the PSAB's objective, but they noted a need for: more promotion of the PSAB to government departments; more information for buyers and consumers on the range of goods and services Aboriginal firms can provide (including improved supplier lists); raising awareness of federal markets among Aboriginal firms; and, maximizing the number of Aboriginal standing offer vendors. A couple indicated that departments should be kept informed of the implications of policy changes on the PSAB, such as purchasing authority changes, in order to understand how to best support Aboriginal businesses. One person suggested Aboriginal benefits requirements could be incorporated within contracts.

3.4.5 Strengths and weaknesses of the PSAB

To gain insight into which aspects of the Strategy are working well and which are not, PSAB program staff and senior procurement officers were asked what the strengths and weaknesses of the PSAB are.

Program staff indicated that the strengths of the PSAB are that it has allowed Aboriginal businesses to access federal contracts and develop partnerships with non-Aboriginal businesses; that it is government wide; that INAC advocates on behalf of Aboriginal businesses; and that it helps create employment, which benefits communities. In terms of weaknesses, program staff indicated that some government employees are still not fully aware of the PSAB; that departments are able to set their own targets, which are often too conservative; that participation is essentially voluntary given that there are no real means of ensuring targets are met; that there remain some ambiguities in the policy (such as those surrounding Aboriginal content); that there is still a lack of Aboriginal capacity in some fields (such as engineering and medical); that Aboriginal businesses may become dependent on PSAB set-aside contracts; and, that small Aboriginal companies may have difficulty competing with larger Aboriginal businesses.

Senior procurement officers indicated that the PSAB's strengths are that it encourages government departments to contract with Aboriginal businesses; that the PSAB website and newsletter are useful; that it has increased interest in federal procurement among Aboriginal businesses and improved their capacity to win contracts; and that the policy is straightforward and easy to understand. As weaknesses, they identified the lack of a pre-qualified supplier inventory, lack of awareness of the PSAB and its developments among certain managers and staff due to insufficient promotion and marketing activity, and lack of data on the impacts of the PSAB.

3.4.6 Lessons learned and suggestions for improving the PSAB

Program staff, senior procurement officers, PSAB coordinators, and Aboriginal businesses were all directly given the opportunity to offer suggestions for improving the PSAB. Material managers were invited to provide any comments they had in addition to the survey questions, and thereby also provided some suggestions as to how the Strategy might be improved.

Program staff suggested implementing an Aboriginal benefits requirement for large contracts. They also indicated the need to clarify the 33% Aboriginal content requirement. Other suggestions included improving communication between suppliers and departments to ensure suppliers are aware of how to access contracting opportunities, establishing a forum through which Aboriginal businesses could exchange information and provide feedback on the PSAB, and establishing mechanisms to limit dependence upon the set-asides (e.g., by establishing a maximum number of contracts firms would be allowed to acquire in this way). A couple suggested linking achievement of departmental targets more closely with the Departmental Performance Reports (DPR) process and performance pay. It was noted that, in the past, federal contractors were not provided with training on the PSAB, but, in the future, training on the PSAB and Comprehensive Land Claim Agreements (CLCA) will be mandatory for designation as a certified purchaser within the federal government, which is expected to improve the effectiveness of the Strategy.

Senior procurement officers suggested undertaking more promotion of the Strategy and of Aboriginal firms with capacity. One person specified there should be more promotion and information sessions of PSAB for program-level staff, who often identify suppliers and can award contracts up to a certain amount. Lost opportunities for Aboriginal suppliers result. When these individuals are unaware of the PSAB, other suggestions included establishing a pre-qualified list of Aboriginal suppliers; implementing better reporting on the impacts of the PSAB; providing more information to suppliers to ensure they understand the bidding process; clarifying ambiguities on the application and interpretation of the PSAB (such as the relationship between PSAB and CLCAs); and, improving the audit process.

Half of PSAB coordinators did not offer any suggestions for improving the PSAB. Suggestions made were: more staff training/awareness-raising (25%); more promotion of Aboriginal businesses (18%); an up to date supplier list (18%), and more audits and compliance activities (11%).

Businesses indicated the need to train departmental staff in the PSAB and encourage them to promote the Strategy more (30%). They also suggested the need for a fairer allocation of contracts (16%), for Aboriginal partners to have more authority (12%), and for the number of projects administered under PSAB to be increase (10%).

Most material managers (65%) did not take the opportunity to provide additional comments on the PSAB. For the most part, comments provided were aligned with those provided by PSAB coordinators and Aboriginal businesses, with 12% noting the need for more data surrounding qualified Aboriginal suppliers and 7% indicating a need for more training.

4.0 Key observations

Rationale

Government stakeholders generally continue to support the PSAB's rationale and expected impacts; however, there is more support for increasing opportunities for Aboriginal firms in other markets and increasing the number of Aboriginal firms competing for federal contracts than for goals such as increasing the value of contracts awarded to Aboriginal firms. This could in part be a reflection of the fact that the number and value of contracts to Aboriginal firms has increased dramatically since the inception of the PSAB, making for a different policy context than when the Strategy was implemented over a decade ago. Interviews with PSAB program staff suggest a desire to move beyond the PSAB by helping Aboriginal firms that have benefited from the Strategy continue to grow without relying solely on contracts set aside under PSAB.

Design and delivery

A number of stakeholders suggested that there are not adequate means of enforcing compliance with the PSAB, given that there are no institutionalized consequences for departments that fail to set aside contracts under the PSAB or who fail to achieve their performance targets.

Some stakeholders also indicated that the eligibility requirements of the PSAB are unclear – particularly the Aboriginal content requirements – and are in need of further clarification. As previously noted, TB submissions are required for any clarification changes to the Strategy. While it is clear that only a minority of firms audited are found to be non-compliant, the consequences of non-compliance are unclear and are not institutionalized, suggesting the potential exists to strengthen enforcement and transparency surrounding the audit process.

Impact/Success

While it is evident that both the number and dollar value of contracts awarded to Aboriginal firms has increased since the inception of the PSAB, there is little data that speaks to the achievement of any of the other expected impacts, particularly the expected indirect impacts. Aboriginal businesses indicated that the PSAB had had a positive impact on such things as their ability to compete for federal contracts and the number and value of contracts awarded to their firm. They indicated that the PSAB had had less of an impact upon investment in their firms and their ability to develop product lines to meet demands. Many were unable to assess the impact of the PSAB upon Aboriginal businesses and Aboriginal communities generally.

PSAB program staff in particular were concerned with the lack of impact data available. They indicated that additional data should be systematically collected to assess the PSAB's impact. This should include the number of jobs and training and apprenticeship opportunities created, as well as whether and to what extent businesses have been able to expand, the number of employees hired who were previously on social assistance, whether and to what extent sales and profit margins increased, and whether the business can now access opportunities in the international marketplace. The survey data suggests that there has been some degree of job creation as a result of the PSAB in a minority of firms.

Lessons learned and future directions

Survey data from material managers and PSAB coordinators indicates that finding Aboriginal firms with the appropriate capacity to undertake particular government contracts remains a significant challenge. Interview and survey data suggests this is due to the lack of an accessible list of pre-qualified suppliers, gaps in capacity in particular fields, such as the medical architectural, engineering, and surveying fields, and the skilled trades.

For promoting Aboriginal business development, PSAB program staff recognized a need to build capacity in certain areas, enhance Aboriginal business capacity, and to see the implementation of an Aboriginal benefits strategy. As noted in Section 3.4.4, as the next step under such a strategy, an Aboriginal benefits requirement would be included on large contracts whereby contractors would be required to demonstrate some commitment to benefiting Aboriginal people in order to be awarded these contracts, through, for example, establishment of a scholarship program, the creation of jobs for Aboriginal people, or a commitment to acquire materials and supplies from Aboriginal vendors. Nevertheless, program staff believe the Aboriginal benefits strategy should be implemented in conjunction with, rather than as a replacement to, the PSAB, as they see the PSAB as an important means through which new and emerging Aboriginal businesses can gain experience. As previously noted, an MC would be required in order to make substantive changes to the PSAB, such as the inclusion of an Aboriginal benefits requirement.

Similar to the evaluation completed in 2002, there was consensus among all parties that there is a need to raise awareness of the PSAB and to actively promote its use.

Management Response / PSAB Action Plan

Evaluation Project Title: Summative Evaluation of the Procurement Strategy for Aboriginal Business (PSAB) for 2001-2006

Evaluation Project #: This evaluation was led by PABP Directorate in consultation with EPMRB. The final report did not contain recommendations. EPMRB developed recommendations based on the findings and conclusions.

Region or Sector: Lands and Economic Development Sector

I approve the Management Response / Action Plan

Original signed by:

Footnotes:

1. Procurement Strategy for Aboriginal Business: Guidelines for Buyers/Government Officials (Contracting Policy Notice 1996-97). (return to source paragraph)
2. Web-Based Guide to Doing Business with the Federal Government, Module 1. (return to source paragraph)
3. ProcurementStrategy for Aboriginal Business: Guidelines for Buyers/Government Officials, op. cit. (return to source paragraph)
4. Procurement Strategy for Aboriginal Business: Guidelines for Buyers/Government Officials (Contracting Policy Notice 1996-97). (return to source paragraph)
5. Procurement Strategy for Aboriginal Business: Guidelines for Buyers/Government Officials (Contracting Policy Notice 1997-6)  (return to source paragraph)
6. Ibid. (return to source paragraph)
7. Ibid. (return to source paragraph)
8. The Procurement Strategy for Aboriginal Business - A Primer for Federal Government Procurement, p. 5. Indian and Northern Affairs Canada. (return to source paragraph)
9. Web-Based Guide to Doing Business with the Federal Government, Module 2. (return to source paragraph)
10. Ibid. (return to source paragraph)
11. To generate this sample, PWGSC provided all documents (originals and amendments) awarded for both the "competitive" and "non-competitive" processes coded against any of the trade agreements associated to the CLCA (Comprehensive Land Claim Agreements), the Set-Aside program and/or which are aboriginal (self) identified suppliers. (return to source paragraph)
12. Note respondents could choose more than one sector. (return to source paragraph)
13. The database included all contracts (originals and amendments) awarded by PWGSC under both "competitive" and "non-competitive" processes coded against any of the trade agreements associated to the CLCA (Comprehensive Land Claim Agreements), the Set-Aside program and/or which are Aboriginal (self) identified suppliers. (return to source paragraph)
14. PSAB - Guidelines for Buyers/Government Officials, Section 1.1.1 (return to source paragraph)
15. PSAB - Guidelines for Buyers/Government Officials, Section 3.2.1 (return to source paragraph)
16. PSAB - Guidelines for Buyers/Government Officials, Section 3.1.1 (return to source paragraph)
17. PSAB - Guidelines for Buyers/Government Officials, Section 3.2.4 (return to source paragraph)
18. PSAB - Policy.  (return to source paragraph)
19. PSAB Performance Report, 2004. Page 16. (return to source paragraph)
20. This response was specified under "other" by 11% of respondents. (return to source paragraph)
21. Department of National Defence, Health Canada, PWGSC, Fisheries and Oceans, Canada Revenue Agency, Correctional Service Canada, Transport Canada, and Canadian Heritage (return to source paragraph)
22. PSAB Performance Report 2004, page 5. (return to source paragraph)
23. The Procurement Strategy for Aboriginal Business — A Primer for Federal Government Procurement, p. 4. Indian and Northern Affairs Canada. (return to source paragraph)
24. Aboriginal Business Procurement Policy and Incentives (Contracting Policy Notice 1996-2). (return to source paragraph)
25. The PSAB website indicates: "Please note that although this new version of the policy is now administered on INAC's web site, it is still a Government of Canada policy, under the authority of the Treasury Board Secretariat." (return to source paragraph)
26. PSAB - Procurement Strategy for Aboriginal Business: Guidelines for Buyers/Government Officials  (Contracting Policy Notice 1997-6) (return to source paragraph)
27. PSAB - Procurement Strategy for Aboriginal Business: Guidelines for Buyers/Government Officials  (Contracting Policy Notice 1997-6) (return to source paragraph)
28. Please note that the data in Table 11 was provided directly by PWGSC, whereas the data in Table 10 was computed by PRA. (return to source paragraph)
29. Respondents could choose more than one industry sector. (return to source paragraph)