Canadian International Trade Tribunal
www.citt-tcce.gc.ca
Common menu bar links
Procurement
Determinations
File No. PR-2001-051
Download full document in MS Word format, compressed - zip (32K) through FTP
Download full document in Adobe Acrobat format (278K) through FTP
Download full document in MS Word format, compressed - zip (32K) through HTTP
Download full document in Adobe Acrobat format (278K) through HTTP
Ottawa, Thursday, May 2, 2002
File No. PR-2001-051
IN THE MATTER OF a complaint filed by DRS Technologies Inc. under subsection 30.11(1) of the Canadian International Trade Tribunal Act, R.S.C. 1985 (4th Supp.), c. 47;
AND FURTHER TO a decision to conduct an inquiry into the complaint under subsection 30.13(1) of the Canadian International Trade Tribunal Act.
DETERMINATION OF THE TRIBUNAL
Pursuant to subsection 30.14(2) of the Canadian International Trade Tribunal Act, the Canadian International Trade Tribunal determines that the complaint is valid in part.
Pursuant to subsections 30.15(2) and (3) of the Canadian International Trade Tribunal Act, the Canadian International Trade Tribunal recommends that the results of the evaluation of all rated requirements of the Request for Proposal, in relation to the consequence of failure factor for the three proposals received, be set aside. Furthermore, it recommends that the Department of Public Works and Government Services and the Department of National Defence re-evaluate the consequence of failure factor, including the key risk categories upon which it depends, for all rated requirements, essential or not, in both the technical and management proposals for the three proposals submitted. The re-evaluation will be conducted strictly, according to the criteria and methodology set out in the Request for Proposal, including the Evaluation Plan. This will require the removal, from the electronic proposal evaluation software, of the consequence of failure categories selected by the subject matter experts, which are inserted in the evaluation screens under the tab "Guidance to Evaluators", and of the estimated scores for the consequence of failure factor. The estimated scores for the consequence of failure factor will, in no way, be used in the re-evaluation. The procurement process will proceed as provided for in the above-mentioned solicitation documents and the Agreement on Internal Trade.
The re-evaluation will be conducted by a new evaluation team that will be composed of members other than those involved in the original evaluation and will exclude the fourth subject matter expert who was involved in determining the estimated scores for the consequence of failure factor prior to the evaluation of proposals.
Pursuant to subsection 30.16(1) of the Canadian International Trade Tribunal Act, the Canadian International Trade Tribunal awards DRS Technologies Inc. its reasonable costs incurred in preparing and proceeding with the complaint.
Patricia M. Close
Patricia M. Close
Presiding Member
Richard Lafontaine
Richard Lafontaine
Member
Ellen Fry
Ellen Fry
Member
Michel P. Granger
Michel P. Granger
Secretary
The statement of reasons will follow at a later date.
|
Date of Determination: |
May 2, 2002 |
|
Date of Reasons: |
May 23, 2002 |
|
Tribunal Members: |
Patricia M. Close, Presiding Member |
|
Richard Lafontaine, Member |
|
|
Ellen Fry, Member |
|
|
Investigation Manager: |
Paule Couët |
|
Investigation Officer: |
Ronald B. Harrigan |
|
Counsel for the Tribunal: |
John Dodsworth |
|
Complainant: |
DRS Technologies Inc. |
|
Counsel for the Complainant: |
Ronald D. Lunau |
|
Phuong T.V. Ngo |
|
|
Interveners: |
Thales Systems Canada |
|
CMC Electronics Inc. |
|
|
Counsel for the Interveners: |
Barbara A. McIsaac, Q.C. |
|
Kris Klein |
|
|
Gregory O. Somers |
|
|
Paul D. Conlin |
|
|
Government Institution: |
Department of Public Works and Government Services |
|
Counsel for the Government Institution: |
David M. Attwater |
Ottawa, Thursday, May 23, 2002
File No. PR-2001-051
IN THE MATTER OF a complaint filed by DRS Technologies Inc. under subsection 30.11(1) of the Canadian International Trade Tribunal Act, R.S.C. 1985 (4th Supp.), c. 47;
AND FURTHER TO a decision to conduct an inquiry into the complaint under subsection 30.13(1) of the Canadian International Trade Tribunal Act.
STATEMENT OF REASONS
COMPLAINT
On December 18, 2001, DRS Technologies Inc. (DRS) filed a complaint with the Canadian International Trade Tribunal (the Tribunal) under subsection 30.11(1) of the Canadian International Trade Tribunal Act. 1 The complaint concerns a procurement (Solicitation No. W8485-01NA20/B) by the Department of Public Works and Government Services (PWGSC), on behalf of the Department of National Defence (DND), for the supply of a communications management system (CMS) for the Canadian Forces' CP-140 Aurora aircraft. The work includes, but is not limited to, the design, building, installation and integration of a new CMS solution for the Aurora aircraft fleet; 2 the modification of various aircraft sub-systems; the provision of a modified systems integration lab and a modified integrated avionics trainer; the provision of an integrated logistics support program; the provision of program management activities to deliver the requirements; and the provision of such necessary data, rights and documentation as are required to ensure the successful delivery and completion of the CMS solution.
DRS alleged that, contrary to several provisions of the Agreement on Internal Trade, 3 the North American Free Trade Agreement 4 and the Agreement on Government Procurement, 5 PWGSC unfairly and improperly disqualified DRS's proposal from further consideration by applying a biased evaluation methodology that included the use of undisclosed preferences, which, along with a disregard for the information contained in the proposal, resulted in its bid receiving unjustifiably low scores. As a result, AVICOM, the product offered by DRS, was unfairly ruled ineligible for further consideration, notwithstanding that it was determined to have met or exceeded all the mandatory technical requirements of Appendix I to the Request for Proposal (RFP).
Specifically, DRS alleged that PWGSC did not adhere to the methodology set out in the RFP to evaluate the consequence of failure. It also alleged that the evaluation process was permeated by an undisclosed preference for "demonstrated flight performance", which tainted the results of the evaluation and rendered them unreliable, and that the evaluation process lacked transparency and predictability with respect to the role of the evaluation software, the "consensus" scoring process and the refusal to provide DRS with the individual evaluation scores relating to its proposal. Finally, DRS alleged that some elements of its proposal were given unjustifiably low scores, reflecting a disregard for the information contained therein, particularly when read as a whole.
DRS requested that the Tribunal immediately issue an order directing PWGSC and DND not to award any contract in relation to this solicitation until the Tribunal determined the validity of the complaint. Furthermore, it requested, as a remedy, that the disqualification of its proposal at stage 3 of the evaluation process be set aside, that its proposal be re-evaluated in an objective and unbiased fashion and that the proposal proceed to stage 4 of the evaluation process in competition with the other proposal(s). DRS also requested its costs incurred in preparing and proceeding with this complaint.
On December 24, 2001, the Tribunal informed the parties that the complaint had been accepted for inquiry, as it met the requirements of subsection 30.11(2) of the CITT Act and the conditions set out in subsection 7(1) of the Canadian International Trade Tribunal Procurement Inquiry Regulations. 6 That same day, the Tribunal issued an order postponing the award of any contract in relation to this solicitation until the Tribunal determined the validity of the complaint. On January 4 and 11, 2002, the Tribunal informed the parties that Thales Systems Canada 7 (Thales) and CMC Electronics Inc. 8 (CMC) had been granted intervener status in the matter.
On January 28, 2002, PWGSC filed a Government Institution Report (GIR) with the Tribunal in accordance with rule 103 of the Canadian International Trade Tribunal Rules. 9 On February 5, 2002, DRS filed a motion with the Tribunal requesting the production of additional documents and an extension of the time period to reply to the GIR. On February 13, 2002, the Tribunal ordered that PWGSC file additional documents relating to the evaluation of proposals. On February 18, 2002, PWGSC filed these additional documents with the Tribunal. On March 1, 2002, DRS, CMC and Thales filed comments on the GIR with the Tribunal. On March 5, 2002, Thales filed comments in response to CMC's comments on the GIR. On March 7, 2002, PWGSC wrote to the Tribunal indicating that it would not request the right to make further submissions and requesting that the Tribunal expedite its inquiry.
On March 13, 2002, the Tribunal requested PWGSC to provide additional information to complete the record. PWGSC filed that additional information with the Tribunal on March 20, 2002. Thales filed comments in response on March 27, 2002, and DRS did likewise on March 28, 2002.
Given that there was sufficient information on the record to determine the validity of the complaint, the Tribunal decided that a hearing was not required and disposed of the complaint on the basis of the information on the record.
PRELIMINARY MATTER
PWGSC submitted that the CMS is classified in Federal Supply Classification (FSC) Group 58 (Communication, Detection, and Coherent Radiation Equipment). It submitted that the RFP is subject to the AIT, but that, pursuant to Article 1001(1)(b) of NAFTA and paragraph 2 of Section A of Annex 1001.1b-1 to NAFTA, the CMS is not covered by NAFTA. More specifically, PWGSC submitted that General Note 1(c) of the Schedule of Canada, Annex 1001.2b to NAFTA provides that NAFTA does not apply to procurements in respect of "contracts respecting FSC 58". It further submitted that, pursuant to Article I(1) of the AGP and Appendix I, Canada, Annex 1 to the AGP, the CMS is not covered by the AGP. More specifically, General Note 1(c) of Appendix I provides that the AGP does not apply to procurements in respect of "contracts respecting FSC 58".
In its comments on the GIR, Thales also submitted that, with respect to procurements conducted for or on behalf of DND, the goods and services related thereto falling under FSC 58 are not covered by NAFTA or the AGP.
In its comments on the GIR, DRS made no submission on the issue of jurisdiction under NAFTA and the AGP. However, it still alleged several breaches of the provisions of the AIT.
The Tribunal agrees with PWGSC and Thales' submission that this procurement is covered by the AIT, but not by NAFTA or the AGP.
PROCUREMENT PROCESS
On October 26, 2000, a Letter of Interest and a draft RFP 10 were posted on MERX. 11 This was followed by a site visit and briefing sessions at Canadian Forces Base Greenwood on November 29 and 30, 2000. Eight companies, including DRS, participated in these events. On January 23, 2001, PWGSC issued the RFP for this solicitation. The RFP closed on April 19, 2001. Three bids were received, one each from DRS, CMC and Thales. The RFP included, at Appendix H, an evaluation plan (EP) 12 with a project evaluation tree attached as Annex A.
The RFP includes the following provisions that are relevant to this case:
[Article 26.2.1]
It is the Bidders' sole responsibility to provide sufficient information to permit a full understanding of what is proposed.
[Article 27.1.1]
The evaluation of the Bidder's offer shall be based solely on the contents of its proposal in response to the RFP, the SOW, the Specifications, and the documents called up therein. Failure to provide sufficient information in any area may result in the assumption of non-compliance in that area.
[Article 27.2.3]
Those bids which are deemed Technically Compliant with the Mandatories will then be evaluated in accordance with the Evaluation Plan included herewith, to determine compliance with the Rated Essential Management & Technical Requirements of the SOW and the Specification, and must receive a pass mark of 70% in each of the Management and Technical requirements in order to be evaluated further.
The following provisions of the EP 13 are relevant to this case:
3.4 STAGE 3 - DETAILED EVALUATION OF RATED REQUIREMENTS, PROPOSAL SCORE ROLL-UPS AND RANKINGS
Compliant proposals from stage 2 will be evaluated in detail against the Rated requirements in terms of compliance, capability and risk. Upon completion of the detailed evaluations (including any re-adjustment for individual score divergences) the ETL [Evaluation Team 14 Leader] shall use the automated features of the Project Evaluation Software (PES)[15] to arrive at scores for all proposals in accordance with the methodology described in this plan. . . . Rated requirements fall into two categories; essential and desirable.
3.9 MANDATORY VERSUS RATED REQUIREMENTS
To safeguard against a proposal not delivering the essential requirements of the CMS work, DND has identified certain items to be Mandatory Requirements. These are identified in Appendix I of the RFP.
A Mandatory Requirement is defined as a requirement that must be met in order for the Bidders' proposals to be further considered for evaluation. Mandatory Requirements are assessed as either compliant or non-compliant prior to scoring and any non-compliant proposals will be eliminated.
Rated requirements fall into two categories - essential requirements and desirable requirements. Rated essential requirements are designated through the use of the word "shall". Rated desirable requirements are designated through the use of the words "should" or "may".
For each proposal, these rated requirements are assessed and scored to determine the proposal's degree of compliance, the bidder's capability to meet the requirement, and any associated risks.
3.10 EVALUATION INTEGRITY & CONSISTENCY
The evaluation process, procedures and interpretation of requirements shall not be changed once the RFP has been released to industry unless an RFP amendment is issued by PWGSC.
6. TECHNICAL AND MANAGEMENT SCORING METHODOLOGY
. . . Each proposal that meets the Mandatory Requirements will be evaluated using the methodology outlined below. Any bidder's proposal that fails to achieve an overall score of 70 percent (against the rated essential requirements only), in each of the CMS SOW and CMS Functional Performance Specification, shall be rejected on the basis that it fails to meet the Technical and Management requirement. The pass or fail calculation shall be done against the rated essential requirements only. The calculation for the final cost per point shall include the rated essential requirements and the rated desirable requirements.
6.1 REVIEW OF MANDATORY REQUIREMENTS
Proposals to meet Mandatory Requirements shall be rated as either "Compliant" or "Non-compliant." Proposals shall be eliminated based on any non-compliance with Mandatory Requirements.
6.2 EVALUATION TREE CRITERIA AND ASSIGNED WEIGHTS
For the Management and Technical portions of the evaluation, this EP is built around an Evaluation Tree (ET). The ET is provided at Annex A.
The Management Proposal is weighted at 35% and includes the Bidders' responses to the SOW requirements in the RFP. The Technical Proposal is weighted at 65% and includes the Bidders' responses to the CMS Functional Performance Specification requirements in the RFP.
The Technical and Management main branch of the ET is broken down into major sub-branches organized along the functional lines found in the project RFP documents. Weights have been designed to ensure appropriate balance and relative importance has been catered to. Note that the weighting factors determine the amount of influence that every proposal element contributes to the overall scores.
6.3 WEIGHTING FACTORS
The numerical weights apportioned to the evaluation criteria (i.e. the evaluated SOW and Specification package) provide a measure of their relative importance in deciding on the winning Proposal. They are computed using the Value Tree method, whereby criteria are organized into a hierarchy tree, with branches and sub-branches. Scoring is then done against the lowest level criteria.
Weights are assigned to branches and sub-branches until all branches have been assigned weights. Branch weights are assigned such that they sum to a normalised value, such as 100. The Final Criteria Weights are calculated as the products of the criterion branch weights times the weights assigned to all of its ancestor branches.
6.4 EVALUATION TECHNICAL AND MANAGEMENT SCORING
This EP presents a standard method for evaluating the Bidders' proposals against each of the Technical and Management branch elements. Element scores will be based on the following factors:a. Compliance Score. The evaluation by the evaluator of the degree which the Bidder states that the requirement will be met, as proposed. It is a measure of how well the proposal meets (or states it meets) the requirement.
b. Capability Score. The evaluation by the evaluator of the Bidders' capability to meet requirements, as proposed. It is a measure of the capability of the Bidder to meet the requirement.
c. Risk Factors. The evaluation by the evaluator of the measure of the risk associated with meeting requirements, as proposed. It is a measure of the level of risk associated with the Bidder meeting the criterion. The key risk categories considered are risks associated with operations, support, or schedule. Other categories of risk may be accommodated.It is the bidder's responsibility to prove compliance and demonstrate capability and the level of risk. If this requires the delivery of supporting documentation in the proposal, the burden of proof is their responsibility, without DND prompting.
6.5 INDIVIDUAL SCORING
All proposals meeting the Mandatory Requirements will be subject to Technical and Management scoring by assigned evaluation team members. Technical and Management scoring will be conducted against each individual evaluation element or criteria. For each proposal, each evaluation element shall be independently assessed in order to assign a Compliance Score, a Capability Score and Risk Score.
6.5.2 Capability Scoring
Capability scoring of each proposal shall be based on the following "word pictures" and scores.
The question that shall be asked is "Based on the claims in the proposal, what is the Bidder's capability to meet this requirement?"
Capability· Previously Proven/Demonstrated Capable 10
· Claimed Capable (with credible approach and reasonable evidence) 8
· Probably Capable (credible approach, limited direct evidence) 7
· Perhaps Capable (insufficient evidence) 3
· Not Likely Capable 1
· . . .
· No Information or Clearly Not Capable 06.6 RISK SCORING
Risk scoring shall consider "Probability" and "Consequence" (of failure) separately and shall be based on the "word pictures" and scores in the following sections.
Using the PES, the evaluators select from a single[ 16 ] Consequence score scale (Major, Very Significant, etc) and enter a single Consequence assessment (i.e., based on operational, support or schedule related consequence categories).
6.6.1 Scoring Probability
When scoring the probability of failure associated with proposal elements, the question that will be asked is:
"Based on the proposal, what is the likelihood that this Bidder will fail to meet this requirement?"
Answer: The likelihood is: Score
Highly Probable - almost certain to fail 1
Very Probable - strongly suspect will fail 3
Probable - good chance will fail 5
Might Happen - may or perhaps will fail 8
Improbable - not likely to fail 10
When scoring "Probability", only "likelihood" should be considered, independent of any other aspect of risk.
6.6.2 Scoring Consequence for Operational Risks
When considering the Operational consequence of failure associated with proposal elements, the question that will be asked is:
"If the Bidder fails to meet this requirement, what would the impact on operations be?"
Answer: The most likely impact would be: Score
Major - Jeopardise Project 1
Very Significant - Potentially Compromise Airworthiness or Capability 3
Significant - Significantly Reduce Capability/Availability 5
Limited - Slightly Reduce Capability/Availability 8
None/Minimal - Minimal to No significant impact 10
6.6.3 Scoring Consequence for Support Risks
When considering the Support related consequence of failure associated with proposal elements, the question that will be asked is "If the Bidder fails to meet this requirement, what would the impact on support be?"
Answer: The most likely impact would be: Score
Major - Make System Generally Unsupportable 1
Very Significant - Major reduction in Supportability 3
Significant - Significant Reduction in Capability/Readiness 5
Limited - Slight Reduction in Supportability 8
None/Minimal - Minimal to No significant impact 10
6.6.4 Scoring Consequence for Schedule Risks
When considering the Schedule related consequence of failure associated with proposal elements, the question that will be asked is "If the Bidder fails to meet this requirement, what would the impact on schedule be?"
The impact would be: Score
Major - Potential to Stop Project or Delay for Years 1
Very Significant - Potential Major Delay (> 1 year) 3
Significant - Potential Major Delay (3-12 months) 5
Limited - Slight Delay (1-2 months) 8
None/Minimal - Minimal Delay (< 1 mo) 10
6.6.5 Scoring Consequence for Programmatic or Other Risks
Consequence scores may be assigned for elements where the consequence of failure cannot easily be categorised as being operational, support or schedule related. For these elements, the ETL may define appropriate word pictures using a similar scoring scale.
6.7 CONSENSUS SCORING
As directed by the ETL, the individual scoring of Compliance, Capability and Risk Factors will be reviewed to achieve consensus by the evaluation team. The ETL is responsible for determining and documenting that consensus scores have been reached. Once consensus is achieved, the rolled up evaluation results are based on these consensus scores.
On October 2, 2001, PWGSC advised DRS, in writing, of its score determined by the PES for the rated essential requirements of the Statement of Work (SOW) and the Functional Performance Specification (the Specification). DRS was also advised that it had not achieved the 70 percent passing mark, as required by the Specification.
In correspondence dated December 4, 2001, PWGSC advised DRS that the score for consequence of failure for each requirement was considered by the individual evaluators and again by the evaluation team when determining consensus scores According to the GIR, individual evaluators actually changed some estimated scores. 17 However, in every case, the evaluation team determined that the consensus score should be the estimated score.
POSITION OF PARTIES
PWGSC's Position
PWGSC maintained that DRS did not fail because of the scoring for consequence of failure but because it was non-compliant with respect to a number of rated essential requirements. Also, according to PWGSC, there was a lack of product maturity of AVICOM, DRS's product, and an absence in DRS's bid of substantial evidence on the development status of the proposed system, which had a negative impact on the capability and probability of scores for consequence of failure.
PWGSC further submitted that estimated scores were used in the PES for assessing the risk associated with the consequence of failure in order to decrease the subjectivity. These estimated scores were decided over several days by a team of four subject matter experts from DND, three of which were subsequently assigned to the evaluation team. Each requirement of the Specification and SOW was reviewed to discuss the impact of a requirement not being met, using the four published consequence of risk categories, i.e. operational, support, schedule and programmatic. The category most impacted was identified and, using the word pictures and definitions for consequence of failure set out in the EP and anticipated proposals from bidders, default scores were assigned to each requirement that best estimated the most likely consequence of failure of a bidder not meeting the requirement. The consequence of risk category preselected by the subject matter experts to estimate the consequence of failure was identified in the PES under the tab "Guidance to Evaluators".
PWGSC added that DND used its best estimate of the consequence of failure to determine the most appropriate passing mark for stage 3 of the evaluation process. Using estimated scores for consequence of failure and minimum acceptable scores for compliance, capability and the probability of risk, DND determined that the minimum passing mark was 70 percent, based on a sensitivity analysis. This scoring would pass a proposal that, at a minimum, was compliant, had claimed to be capable of meeting the requirements and provided evidence to support that claim, and for which the probability of failing the requirement was somewhere between "improbable" and "might happen".
PWGSC submitted that several grounds of complaint raised by DRS, i.e. the use of the PES, the use of consensus scores in the evaluation process and complaints with respect to the evaluation methodology used, were filed with the Tribunal outside the time limit prescribed by section 6 of the Regulations. It submitted that the procurement process does not provide for an accumulation of grievances to be put forward only in the event of an unsuccessful bid (see File No. PR-96-011 18 ) and that the evaluation methodology, including the use of the PES and consensus scoring, was clearly described in the RFP. The RFP was prepublished in October 2000. Therefore, DRS ought to have known of these grounds of complaint at the time at which the RFP was issued in January 2001. It sought no clarification of these matters during the bidding period and did not object to these aspects of the RFP until after its proposal was declared non-compliant.
PWGSC submitted that the evaluation team members did "turn their minds to the merits of the individual proposals" 19 and gave "each proposal meaningful consideration." 20 It argued that the use of estimated scores in the PES was not inconsistent with the evaluation methodology set out in the RFP. Individual evaluators were at complete liberty to award scores for consequence of failure and so was the evaluation team, as a whole, in its consensus scoring. Estimated scores were used on the PES score sheets for practical reasons, because such scores are less affected by the nature of the responses to a requirement and because the consequences of failure will be experienced and best understood by DND.
The evaluation team awarded each bidder the same consensus scores after a detailed review of the individual evaluators' scores. The consensus scores awarded by the evaluation team were identical to the scores estimated by DND's subject matter experts. Although the consensus scores ended up being identical to the estimated scores, there was evidence, according to PWGSC, of evaluation by the individual evaluators, given that two evaluators provided different individual scores for one performance specification. 21
PWGSC asserted that starting with scores that estimated consequence of failure in circumstances where evaluators were at complete liberty to adjust the scores in no way reduced the maximum number of points available to bidders. Contrary to DRS's allegation, PWGSC submitted that 100 percent of the points for consequence of failure were attainable by bidders. However, the evaluation team determined that bidders did not earn 100 percent of these points.
With respect to DRS's allegation that its proposal "handily passes the 70 percent threshold", PWGSC submitted that this assertion is based on faulty arithmetic, which assumes that DRS would score 100 percent of the points available for consequence of failure under its revised scenario. 22 Such could not be the case, since the evaluation team found that DRS's proposal did not warrant all the available points for consequence of failure.
PWGSC added that, since all points available for consequence of failure could have been awarded, a reduction from 61.2270 to 55.4015 points for the rated essential requirements of the Specification, as suggested by DRS, is not justified. DRS's proposal was not assessed using estimated scores for consequence of failure, but rather using the scores awarded by the evaluation team.
PWGSC submitted that removing the consequence of failure factor from the evaluation formula specified in section 6.8 of Appendix H to the RFP would result in an inaccurate assessment of risk. However, if the consequence of failure factor were removed from the evaluation formula and all else remained the same in the formula, the revised passing score would be 77.14. PWGSC asserted that, on this basis, DRS's revised score for the Specification would still be a failing grade.
PWGSC submitted that there were no undisclosed preferences or criteria used in evaluating the proposals. PWGSC agreed with DRS that it was not mandatory for a proposed CMS to be previously flight-proven. However, PWGSC submitted that it was clear from the RFP that DND required a CMS for operational CP-140 aircraft and that the vast majority of the 709 requirements of the CMS were sensitive to DND's needs for an airworthy communications system.
PWGSC submitted that, contrary to DRS's assertion, DND did not require "flight performance in which a CMS system could be proven capable." Rather, 709 requirements of a proposed CMS were evaluated. Many of those requirements were expressly qualified by reference to use in the CP-140 Aurora or, more generally, aircraft. 23 PWGSC noted that, when assessing capability, "Previously Proven/Demonstrated Capable" was the highest assessment that a bidder could achieve. PWGSC submitted that, where an evaluator was assessing the capability of a proposal to meet a specific technical requirement of the CMS for an operational CP-140, it was reasonable for the evaluator to require appropriate evidence of the highest possible capability of meeting the requirement before awarding the highest possible score for capability. Furthermore, PWGSC submitted that it can reasonably be inferred from the EP and the RFP that the evaluators could require evidence from flight performance before awarding the highest score for the capability of certain proposed features of the CMS.
PWGSC submitted that the EP did not dictate the type of evidence that an evaluator could require to establish to his/her satisfaction that a capability was "Previously Proven" or "Demonstrated Capable". It was open to evaluators, based on the "word pictures" included in the EP, to award scores that they considered appropriate in the circumstances.
PWGSC submitted that DRS's allegation of bias in the evaluation of its proposal is completely unfounded. It submitted that DRS's proposal was properly evaluated and that the reasons for its disqualification were clearly exposed in correspondence and subsequent debriefings. PWGSC further submitted that DND's experts that comprised the evaluation team are best qualified to assess DRS's bid and that the Tribunal should not substitute its judgement on scoring for that of the evaluation team (see File No. PR-2000-007 24 ).
PWGSC submitted that the individual evaluators took fully into account the information presented by DRS in its proposal. Similarly, the evaluation team had regard to DRS's complete proposal when determining consensus scoring. That DRS did not achieve a score of 70 percent for the rated essential requirements of the Specification does not support its allegation that the score that it obtained was unfair or that its proposal did not receive the fair, consistent and accurate evaluation promised in the RFP. DRS's score is simply a reflection of its proposal.
In response to a question by the Tribunal, PWGSC submitted that, in assessing risk, the probability of failure must be distinguished from the consequence of failure. It submitted that consequence is a function of a requirement not being met. Given that all bids proposed similar architectures, the scores for consequence of failure for all 709 rated requirements should be similar for all bidders. PWGSC argued that, while different products may result in different scores for probability of failure, the consequence of not meeting a particular requirement should be similar for all bidders with similar architectures.
PWGSC further submitted that, during the pre-evaluation briefing, the evaluation team leader verbally briefed all evaluators on the process by which scores for consequence of failure were estimated and the reason for displaying these data as default scores in the PES. It submitted that the evidence 25 shows that consequence of failure was a live evaluation factor. Evaluators were informed that, for practical reasons, estimated scores for consequence of failure had been imported as default scores in the PES, but that consequence of failure remained an evaluated measure that required their consideration. Therefore, as appropriate, the category of risk had to be changed if, for any reason, their assessment of consequence of failure differed from the estimated scores.
PWGSC further submitted that the numerical weight assigned for each element of the CMS evaluation tree had no bearing on the scores for consequence of failure. It submitted that weight is a measure of importance of a requirement to DND, while consequence of failure is a measure of the impact on the project of a bidder failing to meet a requirement. The assignment of weights and the estimation of scores for consequence of failure were done at different times. Weights were assigned prior to the RFP being issued, while scores for consequence of failure were estimated after the RFP was issued.
With respect to the role of the bidders' initial assessment of risk in evaluating consequence of failure, PWGSC submitted that the initial assessment of risk, required of bidders at article 10.11 of the SOW, describes a process to identify, control and track risks throughout the project life. It submitted that this initial assessment of risk must be contrasted to features that may have been proposed by a bidder to mitigate the consequence of non-compliance with a rated requirement of the RFP. In this context, PWGSC submitted that most of the mitigating features proposed by DRS were geared towards probability of failure and, therefore, did not change the scores for consequence of failure. Moreover, taking into consideration the granularity of the scoring scale for consequence of failure (i.e. possible scores of 1, 3, 5, 8 and 10 only), it submitted that the proposals did not introduce anything significant enough to justify a change to the estimated scores for consequence of failure.
PWGSC indicated that the process by which consensus scoring was achieved varied depending on the individual scores. Where all evaluators had the same score, the score remained unchanged unless new information came to light, e.g. response to a clarification question. Where individual scores differed slightly, there was a discussion until the evaluation team unanimously agreed on a consensus score that reflected the best evaluation of the bidder's response. Where one individual score differed significantly from the rest, the evaluator with the different score explained the basis for the score and, after discussion, a consensus was reached. 26 Where individual scores were divided, there was a discussion until all evaluators reached an agreement.
PWGSC reserved the right to make further submissions on the award of costs.
Thales' Position
Thales submitted that there was clearly an individual assessment of the consequence of failure portion of each proposal.
Concerning DRS's allegation that DND applied undisclosed preferences and criteria in evaluating proposals, Thales submitted that it was clear from the outset, through briefings with the proponents and through the solicitation documents, that preference would be given to an off-the-shelf, non-developmental item with previously proven/demonstrated capability. Therefore, Thales argued, DRS could not reasonably assume that a perfect score of 10 could be granted for any requirement where the system had not yet been proven in an aircraft environment. Comparing its proposed, already flying, Polomar system to DRS's laboratory demonstrated Telephonics, Thales argued that it would be unreasonable to assess the relative maturity of the two systems with the same score.
With respect to the alleged "perverse" effect of the estimated values for consequence of failure, Thales argued that the EP's intent was to award maximum points (10) for a low probability of failure and minimum points (1) for almost certainty of failure. Thales submitted that, because of the way that the EP treats probability and consequence of failure, it is clear how risk (the combination of probability and consequence) fits into the entire picture. The evaluation approach to scoring risk ensured that points awarded for risk converge to zero when either there is certainty of failure or the consequence of failure would be catastrophic. This approach, Thales argued, is not only consistent with the EP, but also very reasonable to evaluate risk.
Thales adopted the submissions of the GIR on the questions of the PES, consensus scoring and the scoring of a number of specific elements of DRS's proposal.
Thales requested its costs incurred in participating in this complaint.
Should the Tribunal find that the complaint is valid, Thales submitted that it would not be reasonable to grant the relief requested by DRS at paragraph 100 of the complaint. According to Thales, it submitted its proposal in response to the RFP on the basis of the published evaluation criteria. Changing the evaluation criteria now, as suggested by DRS, would be unfair and prejudicial to Thales.
DRS's Position
DRS submitted that the GIR confirms that the scoring methodology that was applied with respect to consequence of failure was not in accordance with the terms of the RFP. It submitted that the scores for consequence of failure were estimated before any proposal had been received and on the basis of information that was unknown to bidders and undisclosed to them. DRS added that this unannounced methodology is clearly contrary to the terms of the RFP and, consequently, in breach of Article 506(6) of the AIT. Furthermore, DRS submitted that the method used by PWGSC and DND in evaluating consequence of failure introduced a bias into the scoring in favour of a certain product.
DRS submitted that, contrary to express stipulations in the RFP that the evaluation was to be based "solely" on the contents of the proposals, the estimated scores for consequence of failure were developed on the basis of information that was not contained in the proposals, since the proposals had not yet been received. Alternatively, it submitted that the consensus scores were based on the estimated scores for consequence of failure. This also violates the terms of Section F of the RFP, which specified the documentation upon which the evaluation was to be based. Furthermore, DRS submitted that it is clear from the RFP that the reviewing and scoring of proposals was the responsibility of the evaluation team, not that of DND's "subject matter experts". It submitted that the GIR clearly indicates that the scores for consequence of failure that were assigned to bidders' proposals were not made by the evaluation team on the basis of a review and scoring against the EP, but rather were made ahead of time by subject matter experts using some other undisclosed information.
DRS submitted that PWGSC's suggestion that using estimated scores for consequence of failure was more efficient held only if these scores were not subsequently changed. It submitted that the stated objective of achieving efficiency thus discouraged evaluators from making changes to the predetermined scores. Furthermore, DRS submitted that bias, both real and perceived, was created by starting with predetermined scores. Furthermore, because the predetermined scores were developed by DND's "subject matter experts" over the course of "several days", they would obviously carry a great deal of weight, and they undoubtedly exerted a powerful influence over the evaluators. The facts demonstrate that the evaluation team was greatly influenced by these scores. DRS also submitted that the fact that no estimated scores for consequence of failure were changed during the evaluation argues strongly against the existence of true discretion on the part of evaluators to revise the predetermined scores. Even if the evaluators could theoretically change a score, this did not eliminate the bias and influence created by starting with predetermined scores.
With respect to PWGSC's assertion that the estimated scores for consequence of failure remained unchanged, in part, because "[n]o bidder offered risk-mitigating features that supported a change in [these] scores", DRS submitted that it seems unlikely that all bidders would offer identical risk assessments and identical risk control action. Each bidder offered a different product that carried different risks, and each bidder would expect its individual identification and control of risk to be evaluated independently on the basis of the information in its proposal.
Furthermore, DRS submitted that PWGSC and DND's inability to produce any documentation relating to any briefing that was given to the evaluators on how to score consequence of failure seems to be contrary to the terms of the RFP, which made "[p]romulgating `instructions and guidance to evaluators'" a responsibility of the evaluation team leader. DRS noted that a similar information void existed concerning the development of the estimated scores for consequence of failure.
With respect to the relationship between the estimated scores for consequence of failure and the sensitivity analysis conducted by DND to establish the 70 percent passing mark at stage 3 of the evaluation process, DRS submitted that the 70 percent threshold meant that bidders would have to score very well in other categories to offset the drag of the predetermined scores for consequence of failure, which, in fact, operated as an undisclosed weighting. It submitted that the estimated scores for consequence of failure favoured products with demonstrated flight experience and made it difficult or impossible for products with no such experience to get over the threshold.
Moreover, DRS submitted that, as a result of the biasing of the scoring scheme, which was invisible to bidders, it faced a virtually insurmountable and concealed hurdle right from the outset in meeting the 70 percent threshold. The fact that only Thales, the bidder offering the system that is installed on similar maritime patrol aircraft, passed the 70 percent threshold supports this view. As a result, DRS argued, the outcome of the evaluation was that only the system that DND wanted to purchase, i.e. the one offered by Thales, was able to get over the 70 percent threshold and, because it was the only system to do so, it did not have to be subjected to a cost-per-point comparison with the other bids.
DRS asserted that the modification of the scoring methodology for consequence of failure clearly had a negative impact on its score and that the methodology applied by PWGSC and DND to evaluate proposals against the consequence of failure was improper and contravened Article 506(6) of the AIT for the reasons stated above.
Concerning consensus scoring, DRS submitted that, whatever consensus scoring process was developed and applied, it resulted in a breakdown of the evaluation methodology described in the RFP. DRS submitted that the RFP constructed a very elaborate mathematical procedure for achieving numbers representing individual scores. Special software (the PES) was tailored specifically for this project and used by evaluators to track and record these individual scores. Then, by some still unknown process, the numbers resulting from this elaborate individual exercise were replaced by other numbers derived from a "consensus" scoring process about which virtually nothing is known.
The consensus scores, DRS submitted, were produced through discussions of an undisclosed nature, which did not take into account the individual scores in a predictable manner. There was no mathematical relationship between the numbers representing the individual scores and the numbers representing the consensus scores. In this context, DRS argued that a procedure that replaces one set of numbers in an unknown and unpredictable manner with another set of numbers and that may change a passing score into a failing one is unfair to bidders and contrary to the objective of the AIT of bringing greater openness and fairness to the procurement process.
Concerning the timeliness of its concerns with consensus scoring and the PES, DRS submitted that it is not challenging the fact that consensus scoring and the PES were used as announced in the RFP. Rather, it is disputing the manner in which these were applied, which remains largely unknown. DRS submitted that, generally, its grounds of complaint do not deal with the methodology set out in the RFP or the terms thereof, but rather relate to the manner in which the evaluation was carried out and the actions of the evaluators and the methodology that they applied. Therefore, DRS submitted, these aspects of its complaint are not out of time.
DRS further submitted that the GIR misrepresents the development status of AVICOM and the capabilities of DRS, as these were set out in its proposal. The GIR attempts to discredit AVICOM's viability and DRS's ability to deliver a timely solution. DRS submitted that this point of view is not supportable by a factual review of its proposal. Moreover, there was nothing in the RFP to indicate that only systems with demonstrated flight experience would receive the maximum 10 points for capability. Furthermore, DRS argued that the scoring scheme in the EP does not provide for proposals to receive "somewhere between" two point scores and that the PES also did not seem to allow for this. Therefore, in reality, somewhere between 8 and 10 points means 10 points.
With respect to the issue of the lack of "substantial" evidence in its proposal, DRS submitted that no amount of evidence that it could offer would overcome the hidden bias against AVICOM. The failing is not in the evidence that it offered in its proposal, but in the hidden preference that DND designed into the scoring system. DRS submitted that the EP requires that "reasonable evidence" be used in connection with the assessment of the proposed solutions' capabilities and not "substantial evidence" as indicated in the GIR. DRS submitted that it is unfair to take refuge behind undefined qualifications such as "substantial" evidence because this can be as much or as little as PWGSC and DND want it to be, a standard that lacks transparency.
Concerning the submissions in the GIR that AVICOM "lacked maturity" 27 and the lack of information in DRS's proposal on the development status of the system that it proposed, 28 DRS submitted that these assertions are not supported by the facts. It submitted that the same comment applies to PWGSC's and DND's critical assessments of its knowledge and experience with respect to avionics packages generally and the Aurora aircraft in particular 29 and how the AVICOM development schedule would be phased in the tight CMS schedule. 30 DRS submitted that these assertions cannot be fairly supported in light of the evidence included in its proposal.
In its response to the questions asked by the Tribunal and the answers received by PWGSC, DRS submitted the following.
The detailed information submitted by PWGSC in relation to the four requirements for which DRS's proposal was found non-compliant indicates that the consensus scoring resulted in many inexplicable changes to DRS's scoring by individual evaluators in a way that was neither transparent nor predictable. In addition, in a number of cases, PWGSC mischaracterized the information in DRS's proposal. In many cases, the majority individual scores were not reflected in the consensus scores. In multiple instances, initial passing scores were downgraded into failing scores during that process. On this basis, DRS argued that one cannot conclude that consensus scoring produced a fair and reliable evaluation of its proposal.
With respect to the alleged basis, i.e. predicted architecture, used by the subject matter experts to determine the estimated scores for consequence of failure, DRS submitted that, because many variables play in designing the architecture of a complex system such as the CMS, the outcome selected in the end by each bidder cannot be easily anticipated. It submitted that the fact that one of the evaluators who attempted to change a predetermined score for consequence of failure was one of the subject matter experts supports DRS's position that it was inappropriate to start the evaluation with such scores. If bidders' architectures were as predictable as PWGSC says they were, DRS questions why it was that one of the experts found it necessary to propose a change to the estimated scores for consequence of failure.
Finally, DRS agreed with paragraphs 34 to 79 in CMC's complaint on the same procurement relating to the scoring for consequence of failure, unpublished evaluation weighting factors, improper calculations and the effects of the undisclosed values for consequence of failure. 31
CMC's Position
CMC filed the submissions that it made in response to the GIR in File No. PR-2001-052, 32 including a table of concordance, in support of DRS's position.
TRIBUNAL'S DECISION
Subsection 30.14(1) of the CITT Act requires that, in conducting an inquiry, the Tribunal limit its considerations to the subject matter of the complaint. 33 Furthermore, at the conclusion of the inquiry, the Tribunal must determine whether the complaint is valid on the basis of whether the procedures and other requirements prescribed in respect of the designated contract have been observed. Section 11 of the Regulations further provides that the Tribunal is required to determine whether the procurement was conducted in accordance with the applicable trade agreements, in this instance, the AIT.
DRS alleged that PWGSC and DND, acting in breach of Articles 501 and 506(6) of the AIT, improperly disqualified its proposal from further consideration at stage 3 of the evaluation process.
Pursuant to Article 501 of the AIT, the purpose of Chapter Five of the AIT is to establish a framework that will ensure equal access to procurement to all Canadian suppliers. Against this backdrop, Article 506(6) provides, in part, that "[t]he tender documents shall clearly identify the requirements of the procurement, the criteria that will be used in the evaluation of bids and the method of weighting and evaluating the criteria."
The Tribunal must determine whether DRS's proposal was properly disqualified from further consideration at stage 3 of the evaluation process. This will entail considering whether, pursuant to Article 506(6) of the AIT, PWGSC and DND used and applied properly, in evaluating DRS's proposal, the evaluation criteria and methodology set out in the RFP (which includes the EP).
The Tribunal finds that the estimated scores for consequence of failure used as default scores in the evaluation of DRS's proposal are tantamount to weights. As such, the default scores introduced a material change in the evaluation criteria and methodology from those communicated in the solicitation documents, which was not transparent to bidders. This is contrary to Article 506(6) of the AIT.
In reaching this conclusion, the Tribunal carefully considered PWGSC and Thales' submission that individual and consensus scores for consequence of failure were awarded during the evaluation according to the process described in the EP and were not set in advance.
In the Tribunal's opinion, PWGSC and Thales' submission is not sufficient to explain the perfect match between the default scores for each of the 709 rated essential and desirable requirements of the SOW and the Specification for the three proposals and their subsequent scores by the evaluation team. Moreover, there also appears to be a perfect match between the preselected and subsequent evaluator-selected key risk categories.
In the Tribunal's opinion, it is improbable that this perfect match is a coincidence, particularly when considered in light of PWGSC and DND's explanation that the default scores were developed by subject matter experts as a practical guide to evaluators to remove the subjectivity involved in assessing consequence of failure and to create a more efficient human-machine interface. Given the fact that PWGSC and DND intended the default scores to remove subjectivity indicates to the Tribunal that they intended the default scores to influence the scoring. Although there is no evidence that the evaluators made a conscious decision to duplicate the default scores, they did in fact duplicate the default scores. In the Tribunal's view, it would be difficult for an evaluator to avoid being influenced by the default scores particularly because the default scores were clearly identified in the PES score sheets under "Evaluator", as were the risk categories under the tab "Guidance to Evaluators". Moreover, three of the persons who prepared the default scores were members of the evaluation team.
The evidence shows that not only did the six evaluators come up with the same score for consequence of failure as the default score, but they also made the same determination with respect to the preselected consequence of failure category, upon which these scores were based for each of the 709 rated requirements. Their "scoring" of these requirements, in evaluating the three proposals, was identical in all respects except for two instances. In the Tribunal's view, this improbable coincidence of scoring means that the default scores were, in effect, tantamount to weights.
The Tribunal also finds that the effect of estimated scores for consequence of failure reduced the total number of points available and, as a result, modified the calculation of the percentage score at stage 3 of the evaluation process. Furthermore, in the Tribunal's opinion, the effect of using the default scores was to alter, in a non-transparent manner, the evaluation methodology as it was known to DRS at the time of bidding.
With respect to the use of evidence of flight performance in evaluating the capability of the system proposed by DRS, the Tribunal finds that this ground of complaint has no merit.
In the Tribunal's opinion, the RFP did not make it mandatory that the proposed CMS be flight-proven. However, this does not mean that evidence of flight performance could not be used in rating certain aspects of the response to the RFP, as was done in this instance. In the Tribunal's opinion, the real question is whether, under the RFP, evaluators could reasonably require evidence of flight performance as a prerequisite to attributing the highest rating for capability, particularly for those requirements that cannot be completely tested outside an operational aircraft.
The Tribunal is satisfied that the RFP made it abundantly clear that this procurement is for a CMS onboard operational aircraft, i.e. the CP-140 Aurora. As well, the RFP indicated DND's preference for off-the-shelf, non-developmental items. It is clear, in reading the SOW and the Specification, that a significant number of the 709 rated requirements are sensitive to the operational environment of an aircraft. Moreover, the EP clearly stated, at section 6.5.2, that, in order to obtain the maximum score of 10 for capability for any rated requirement, a bidder had to show that the item was "Previously Proven/Demonstrated Capable". Under these conditions, the Tribunal is satisfied that the capability criterion and the manner in which it would be assessed were clearly stated in the RFP and that the evaluators did not breach the provisions of the RFP when they required evidence of flight performance before awarding a capability score of 10 to certain requirements sensitive to the operational conditions of an aircraft.
With respect to the question of the use of the PES and consensus scoring in evaluating DRS's proposal, the Tribunal finds that this ground of complaint was filed outside the 10-working-day time limit prescribed by subsection 6(1) of the Regulations and is, therefore, late.
The Tribunal is of the view that the RFP clearly indicated that the PES and consensus scoring would be used during the evaluation of proposals. It is satisfied that the RFP and EP indicated, albeit summarily, how the PES and consensus scoring would be used during the evaluation of proposals. If DRS was of the view that these provisions were wanting, it should have raised the matter with PWGSC or complained to the Tribunal within 10 working days from the date on which the RFP was issued. This was not done and, accordingly, the Tribunal does not have the jurisdiction to address the merits of this ground of complaint.
With respect to the scoring of four specific requirements in DRS's proposal and the alleged unjustifiably low scores that these requirements received, the Tribunal finds that this ground of complaint is without merit.
In reaching this conclusion, the Tribunal has carefully reviewed the detailed information provided by PWGSC with respect to the four items at issue. Subject to the Tribunal's determination on the scoring for consequence of failure, as set out above, there is no evidence to indicate that these four items were not evaluated according to the criteria and methodology set out in the solicitation documents or that the evaluators did not apply their minds to these items at the time of the evaluation. In the absence of a breach of the evaluation procedures, as alleged, the Tribunal will defer to the judgement of the evaluators. Furthermore, the Tribunal finds that PWGSC's claim that DRS would have failed, because it failed these four rated essential requirements, is unsubstantiated. First, one of the four items relates to a requirement of the SOW. PWGSC stated that DRS passed the SOW. Second, nowhere in the RFP is there a reference to a requirement to pass all rated essential requirements; rather, the requirement is to obtain a 70 percent average. Third, DRS had already passed stage 2, "Mandatory Technical & Management Requirements Screening". Article 27.2.3 of the RFP and article 3.9 and section 6 of the EP all clearly state the difference between the mandatory requirements of stage 2, which are assessed as either compliant or non-compliant prior to scoring, and the rated essential and desirable requirements of stage 3, which are given a score. It is the scores of the items at stage 3 that are at issue in this complaint. The "Mandatory Requirements" within the meaning of the RFP or the EP have already been addressed at stage 2 and are no longer at issue.
The Tribunal finds that DRS's allegation that DND and PWGSC applied a biased evaluation methodology to achieve a specific predetermined outcome, that is, to favour Thales' bid as the only one with demonstrated flight experience, is without foundation.
In the Tribunal's opinion, there is no evidence on the record to support this allegation. The Tribunal notes that, although the probability of failure is more sensitive to the choices made by bidders, the weighting effect of the default scores for consequence of failure was the same for Thales and for DRS. While the evaluation with respect to the default scores for consequence of failure was not transparent, there is no evidence that it was discriminatory. Moreover, as indicated above, the RFP informed bidders that this solicitation was for operational aircraft and encouraged them to offer off-the-shelf, non-developmental items and solutions in determining the best value to the Crown. That DRS may, due to corporate circumstances, find it more demanding than some other potential bidder to be successful under the terms and conditions of the RFP does not, in the Tribunal's opinion, support DRS's allegation of bias.
In conclusion, the Tribunal finds that, due to the inappropriate use of the estimated scores for consequence of failure, DRS's proposal was improperly disqualified from further consideration.
In determining the appropriate remedy in the circumstances, the Tribunal considered the prejudice caused to DRS and other bidders by this improper evaluation. In preparing their bids, the bidders relied on the criteria and evaluation methodology set out in the RFP. If they had known how consequence of failure would be evaluated, they might have submitted different bids. Consequently, the Tribunal is of the view that the evaluation process should be corrected so that bids are evaluated in compliance with the criteria and evaluation methodology set out in the RFP. Given that the Tribunal is of the view that the use of estimated scores for consequence of failure based upon preselected risk categories is the only valid ground of complaint in this instance, it recommends that only the consequence of failure and the risk categories upon which each score depends be re-evaluated. This re-evaluation is to be done for all 709 rated requirements for the three proposals, in accordance with the criteria and methodology set out in the RFP. Based upon the results of this re-evaluation, a successful bidder will be identified in accordance with the relevant provisions of the RFP and the AIT.
DETERMINATION OF THE TRIBUNAL
In light of the foregoing, the Tribunal determines that the procurement was not conducted in accordance with the provisions of the AIT and that the complaint is valid in part.
Pursuant to subsections 30.15(2) and (3) of the CITT Act, the Tribunal recommends that the results of the evaluation of all rated requirements of the RFP, in relation to the consequence of failure factor for the three proposals received, be set aside. Furthermore, it recommends that PWGSC and DND re-evaluate the consequence of failure factor, including the key risk categories upon which it depends, for all rated requirements, essential or not, in both the technical and management proposals for the three proposals submitted. The re-evaluation will be conducted strictly, according to the criteria and methodology set out in the RFP, including the EP. This will require the removal, from the electronic proposal evaluation software, of the consequence of failure categories selected by the subject matter experts, which are inserted in the evaluation screens under the tab "Guidance to Evaluators", and of the estimated scores for the consequence of failure factor. The estimated scores for the consequence of failure factor will, in no way, be used in the re-evaluation. The procurement process will proceed as provided for in the above-mentioned solicitation documents and the AIT.
The re-evaluation will be conducted by a new evaluation team that will be composed of members other than those involved in the original evaluation and will exclude the fourth subject matter expert who was involved in determining the estimated scores for the consequence of failure factor prior to the evaluation of proposals.
Pursuant to subsection 30.16(1) of the CITT Act, the Tribunal awards DRS its reasonable costs incurred in preparing and proceeding with the complaint.
1 . R.S.C. 1985 (4th Supp.), c. 47 [hereinafter CITT Act].
2 . DND operates a fleet of 18 CP-140 Aurora aircraft as a long-range maritime patrol platform for surface and undersea surveillance roles.
3 . 18 July 1994, C. Gaz. 1995.I.1323, online: Internal Trade Secretariat <http://www.intrasec.mb.ca/eng/it.htm> [hereinafter AIT].
4 . 32 I.L.M. 289 (entered into force 1 January 1994) [hereinafter NAFTA].
5 . 15 April 1994, online: World Trade Organization <http://www.wto.org/english/docs_e/legal_e/final_e.htm> [hereinafter AGP].
6 . S.O.R./93-602 [hereinafter Regulations].
7 . Formerly Thompson-CSF.
8 . Formerly BAE Systems Canada Inc.
9 . S.O.R./91-499.
10 . During November 2000, the draft RFP was reviewed, before its release, for consistency and conformity with applicable rules by BMCI Consulting Inc. (BMCI). (See complaint, tab N.)
11 . Canada's Electronic Tendering Service.
12 . The EP was designed by Electronic Warfare Associates.
13 . Six amendments to the RFP were issued, including amendment No. 001, which, inter alia, amended the EP.
14 . According to the GIR, the evaluation team consisted of several individuals, including a leader. As set out in section 4.5 of the EP, the ETL was responsible, in part, for resolving any extreme scores, determining the official consensus regarding evaluators' scores for each proposal evaluation element and identifying high-risk areas. According to the GIR, four evaluators came from the "Project Management Office for the CP-140 Aurora Incremental Modernization Project"; and two evaluators came from "operational CP-140 squadrons at 14 Wing Greenwood N.S." without previous exposure to the bidders, the products or the RFP. Also according to the GIR, airworthiness, quality assurance and DTSES/TEMPEST representatives were utilized as required. Each evaluator had an equal voice in the evaluation process. Individual evaluators started their evaluation of different bids and of different sections of the bids. Discussions among evaluators were to be limited during the individual scoring phase of the evaluation to ensure that all proposals were treated consistently and equally, without preferential treatment or bias, as required by section 3.10 of the EP. Each evaluator had a computer with a unique password that prohibited access, except by the leader. Once all the individual scores were awarded, the evaluators were brought together to decide on consensus scores. The evaluation team discussed individual scores for a requirement before agreeing on a consensus score, and this was done for every rated requirement in the RFP.
15 . According to the GIR, the PES was initially developed for an earlier DND solicitation. It was modified and used for bid evaluations on several other government projects in accordance with the project's unique procurement strategies and evaluation plan. The PES has the equation used to calculate normalised scores and the published weights for each rated requirement imbedded in its software. The PES database is an efficient, single repository for all information relating to the evaluation activities. The PES reflects the technical and management criteria contained in the EP and the RFP. In addition to ensuring control and security of sensitive evaluation results, the PES "guides the evaluation team scoring using a range of compliance, capability and risk factors in a consistent manner, by using a systematic and disciplined approach". The PES supports both subjective and objective evaluations of proposals by using criteria and weights defined by the project management office. It eliminates human errors in manually calculating rolled-up scores to compare proposals.
16 . Amendment No. 001 to the RFP reads, in part:
Question 18
Section 6.6 of the evaluation plan (29395 v3 - CMS EVALUATION PLAN) notes separate risk consequence scores will be assessed for Operational, Support, Schedule, and Programmatic Risks. Section 6.8.1 does not explain how the single value for Consequence that appears in the Criterion Risk Score equation is calculated from the four separate Consequence scores. Could the mathematical equation used to calculate Consequence as a function of Operational, Support, Schedule, and Programmatic Risks, be provided?
Answer 18
The intent is to apply the most applicable and significant of the four Risk elements. Thus, only one of the four consequences will be used to score each requirement.
17 . PES Screen, performance specification 3.3.21.9, GIR (confidential), Exhibit 4.
18 . Re Complaint Filed by Corel Corporation (21 November 1996) (CITT).
19 . Complaint, para. 58.
20 . Ibid., para. 56.
21 . GIR, Exhibit 4.
22 . DRS would keep all the points that it scored for consequence of failure under the EP, specified in the RFP, which it deems to be 100 percent of the points available for consequence of failure under its revised scenario.
23 . See sections 3.3.22, 3.6.1, 3.6.14.2.1 and 3.6.14.11.1 of the Specification and sections 1.2.2 and 3.2.1 of the SOW.
24 . Re Complaint Filed by FMD International (22 August 2000) (CITT) at 4.
25 . PWGSC's submissions, 18 February 2002, Document 1 at. 22, 27-29.
26 . PWGSC's response, 20 March 2002, Question 6 (c).
27 . DRS's comments on the GIR, paras. 83-88.
28 . Ibid., paras. 101, 102.
29 . Ibid., paras. 90-99.
30 . Ibid., para. 100.
31 . DRS's comments on the GIR, para. 40.
32 . Re Complaint Filed by CMC Electronics (2 May 2002) (CITT), where CMC's submissions are reported at length in the statement of reasons.
33 . This determination deals with DRS's complaint only. However, CMC, which also filed a separate complaint on the same procurement, was granted intervener status in this matter, and the submissions that it made have been considered by the Tribunal.
[Table of Contents]
Initial publication: May 23, 2002
