STATE OF FLORIDA

DIVISION OF ADMINISTRATIVE HEARINGS

NATIONAL COMPUTER SYSTEMS, INC., )

)

Petitioner, )

)

vs. ) Case No. 99-1226BID

)

DEPARTMENT OF EDUCATION, )

)

Respondent, )

)

and )

)

HARCOURT BRACE EDUCATIONAL )

MEASUREMENT, )

)

Intervenor. )

)

RECOMMENDED ORDER

Pursuant to notice, Don W. Davis, a duly-designated Administrative Law Judge of the Division of Administrative Hearings, held a formal hearing in the above-styled cause on April 14-15, 1999, in Tallahassee, Florida.

APPEARANCES

For Petitioner: Karen D. Walker, Esquire

Holland and Knight, LLP Post Office Drawer 810 Tallahassee, Florida 32302

Mark D. Colley. Esquire Holland and Knight, LLP Suite 400

2100 Pennsylvania Avenue, Northwest Washington, D.C. 20037

For Respondent: Charles S. Ruberg, Esquire

Department of Education The Capitol, Suite 1701

Tallahassee, Florida 32399-0400

For Intervenor: Paul R. Ezatoff, Jr., Esquire

Christopher B. Lunny, Esquire Katz, Kutter, Haigler, Alderman,

Bryant and Yon, P.A.

106 East College Avenue, Suite 1200 Tallahassee, Florida 32301-7741

STATEMENT OF THE ISSUES

The primary issue is whether the process used by the Department of Education (Department) for evaluating and ranking the proposals submitted in response to Request For Proposal (RFP) 99-03 for the Florida Comprehensive Assessment Test (FCAT) administration contract was contrary to the provisions of the RFP in a way that was clearly erroneous, contrary to competition, arbitrary, or capricious.

PRELIMINARY STATEMENT

On or about November 6, 1998, the Department issued RFP 99- 03 which provided for the award of two contracts, one to a "development contractor" and one to an "administration contractor." Two proposals were submitted to be the administration contractor. The proposals were from Harcourt Brace Educational Measurement (Harcourt) and National Computer Systems, Inc. (NCS).

The Department gave notice of its intended decision to award the administration contract to Harcourt. NCS filed its notice of protest and formal written protest (Amended Formal Written Protest/Petition for Formal Proceeding) pursuant to Section 120.57(3), Florida Statutes. The matter was referred to the

Division of Administrative Hearings (Division), commencing this proceeding. Harcourt’s Motion to Intervene was granted. The award of the development contract is not at issue.

The following witnesses were called to testify through direct and cross-examination: Richard Parizek; Thomas Fisher, Ph.D.; Mark Heidorn, Ph.D.; William Bramlett; Karen Bennett; Cornelia Orr, Ph.D.; and Rob Lucas. The parties stipulated to 16 Joint Exhibits. NCS’s Exhibits numbered 1 through 11 were admitted into evidence. Harcourt’s Exhibits numbered 3 through 6 were admitted into evidence.

Each party timely submitted Proposed Recommended Orders which have been utilized in the preparation of this Recommended Order to the extent possible.

FINDINGS OF FACT

1. The RFP for the FCAT describes a five stage process for evaluating proposals. In Stage I, the Department’s Purchasing Office determined whether a proposal contained certain mandatory documents and statements and was sufficiently responsive to the requirements of the RFP to permit a complete evaluation. Stage II involved the Department’s evaluation of a bidder’s corporate qualifications to determine whether the bidder has the experience and capability to do the type of work that will be required in administering the FCAT. Stage III was the Department’s evaluation of a bidder’s management plan and production proposal. In Stage IV, the Department evaluated a bidder’s cost proposal. Stage V

   involved the ranking of proposals based on points awarded in Stages II-IV.

2. If a proposal did not meet the requirements at any one stage of the evaluation process, it was not to be evaluated in the following stage. Instead, it was to be disqualified from further consideration.

3. Stages II and III of the evaluation process were conducted by an evaluation team comprised of six Department employees: Dr. Debby Houston, Ms. Lynn Joszefczyk, Dr. Peggy Stillwell, Dr. Cornelia Orr, Dr. Laura Melvin, and Ms. Karen Bennett. Dr. Thomas Fisher, head of the Department’s Assessment and Evaluation Services Section, and Dr. Mark Heidorn, Administrator for K-12 Assessment Programs within the Department’s Assessment and Evaluation Services Section, served as non-voting co-chairs of the evaluation team.

4. The focus of this proceeding is Stage II of the evaluation process addressing a bidder’s corporate qualifications.

   RFP Provisions Regarding Corporate Qualification

   
   

5. The FCAT administration contractor will be required to administer tests to approximately one and a half million students each year in a variety of subject areas at numerous grade levels. The FCAT program involves a complex set of interrelated work activities requiring specialized human resources, technological systems and procedures. The FCAT must be implemented annually within limited time periods. The FCAT administration contractor

   must meet critical deadlines for the delivery of test materials to school districts and the delivery of student scores prior to the end of the school year.

6. In developing the RFP, the Department deliberately established a set of minimum requirements for corporate qualifications that a bidder was to demonstrate in order for its proposal to be eligible for further evaluation.

7. The purpose of the RFP’s minimum corporate qualifications requirements was to limit bidding to qualified vendors who have demonstrated prior experience in successfully administering large-scale assessment projects like the FCAT, thereby providing the Department with some degree of assurance that the winning bidder could successfully administer the FCAT.

8. The instructions to bidders regarding the minimum requirements for corporate qualifications are contained in RFP Section 10, which gives directions on proposal preparation.

9. Section 10.1, which lists certain mandatory documents and statements to be included in the bidder’s proposal, requires that a transmittal letter contain "[a] statement certifying that the bidder has met the minimum corporate qualifications as specified in the RFP." These "minimum corporate qualifications" are set forth in RFP Appendix J.

10. RFP Section 10.2 identifies what a bidder is required to include in its proposal with respect to corporate qualifications. The first paragraph of Section 10.2 directs a

    bidder generally to describe its qualifications and experience performing tasks similar to those that it would perform in administering the FCAT, in order to demonstrate that the bidder is qualified where it states:

    Part II of a bidder’s proposal shall be entitled Corporate Qualifications. It shall provide a description of the bidder’s qualifications and prior experience in performing tasks similar to those required in this RFP. The discussion shall include a description of the bidder’s background and relevant experience that qualifies it to provide the products and services required by the RFP.

11. RFP Section 10.2, however, is not limited to a directive that qualifications and past experience be described generally. Instead, Section 10.2, also communicates, in plain and unambiguous terms, that there are specific minimum corporate qualifications a bidder must demonstrate:

    The minimum expectations for corporate qualifications and experience are shown in Appendix J. There are two separate sets of factors, one set of eight for the developmental contractor and another set of nine for the administration contractor.

    Bidders must demonstrate their Corporate Qualifications in terms of the factors that are applicable to the activities for which a bid is being submitted -- development or administration. For each criterion, the bidder must demonstrate that the minimum threshold of experience has been achieved with prior completed projects. (Emphasis

    added.)

12. Moreover, Section 10.2 singles out for emphasis, in relation to the administration component of the RFP, the importance

    placed on a bidder’s ability to demonstrate experience processing a large volume of tests:

    The [bidder’s prior completed] projects must have included work tasks similar to those described herein, particularly in test development or processing a comparable number of tests. The bidder will provide a description of the contracted services; the contract period; and the name, address, and telephone number of a contact person for each of the contracting agencies. This description shall (1) document how long the organization has been providing similar services; (2) provide details of the bidder’s experience relevant to the services required by this RFP; and (3) describe the bidder’s other testing projects, products, and services that are similar to those required by this RFP. (Emphasis added.)

    The Department thus made clear its concern that bidders demonstrate experience with large-scale projects.

13. RFP Appendix J sets forth nine different criteria (C1 through C9) for the administration contractor. As stated in RFP Section 10.2, "[f]or each criterion, the bidder must demonstrate that the minimum threshold of experience has been achieved with prior completed projects . . . ." (emphasis added).

14. Appendix J contains a chart which lists for each criterion: (1) a summary of the related FCAT work task, (2) the detailed criteria for the bidder’s experience related to that work task, and (3) the necessary documentation a bidder must provide.

15. Criterion C4 and Criterion C6 include work tasks that involve the use of image-based scoring technology. C4 and C6 are

    the only corporate qualifications criteria at issue in this proceeding.

    RFP Provisions Involving Corporate Qualifications for Image-Based Scoring

    
    

16. "Handscoring" is the test administration activity in which open-ended or performance-based student responses are assessed. This practice involves a person reading something the student has written as part of the test, as distinguished from machine scoring multiple choice responses (i.e., the filled-in "bubbles" on an answer sheet). There are two types of handscoring:

    (1) paper-based handscoring, and (2) image-based handscoring. Paper-based handscoring requires that a student response paper be sent to a reader, who then reviews the student’s response as written on the paper and enters a score on a separate score sheet. Image-based handscoring involves a scanned image of the student’s response being transmitted to a reader electronically. The student’s response is then projected on a computer screen, where the reader reviews it and assigns a score using the computer.

17. The RFP requires that the reading and math portions of the FCAT be handscored on-line using imaging technology beginning with the February 2000 FCAT administration. The RFP provides that the writing portion of the FCAT may be handscored using either the paper-based method or on-line imaging technology during the February 2000 and 2001 FCAT administrations. However, on-line image-based scoring of the writing portion of the FCAT is required for all FCAT administrations after February 2001.

18. An image-based scoring system involves complex computer technology. William Bramlett, an expert in designing and implementing large-scale imaging computer systems and networks, presented unrefuted testimony that an image-based scoring system will be faced with special challenges when processing large volumes of tests. These challenges involve the need to automate image quality control, to manage the local and wide area network load, to assure adequate server performance and storage requirements, and to manage the work flow in a distributed environment. In particular, having an image-based scoring system process an increasing volume of tests is not simply a matter of adding more components. Rather, the system’s basic software architecture must be able to understand and manage the added elements and volume involved in a larger operation.

19. According to Bramlett, there are two ways that the Department could assess the ability of a bidder to perform a large- scale, image-based scoring project such as the FCAT from a technological perspective: (1) have the bidder provide enough technological information about its system to be able to model or simulate the system and predict its performance for the volumes involved, or (2) require demonstrated ability through completion of prior similar projects.

20. Dr. Mark Heidorn, Administrator for Florida’s K-12 Statewide Assessment Programs, was the primary author of RFP Sections 1-8, which describe the work tasks for the FCAT -- the

    goods and services vendors are to provide and respond to in their technical proposals. Dr. Heidorn testified that in the Department’s testing procurements involving complex technology, the Department has never required specific descriptions of the technology to be used. Instead, the Department has relied on the bidder’s experience in performing similar projects. Thus, the RFP does not specifically require that bidders describe in detail the particular strategies and approaches they intend to employ when designing and implementing an image-based scoring system for FCAT. Instead, the Department relied on the RFP requirements calling for demonstrated experience as a basis to understand that the bidder could implement such an image-based scoring system.

21. Approximately 717,000 to 828,000 student tests will be scored annually by the FCAT administration contractor using imaging technology. The RFP, however, does not require that bidders demonstrate image-based scoring experience at that magnitude. Instead, the RFP requires bidders to demonstrate only a far less demanding minimum level of experience using image-based scoring technology. Criterion C4 and Criterion C6 in Appendix J of the RFP each require that a bidder demonstrate prior experience administering "a minimum of two" assessment programs using imaged- based scoring that involved "at least 200,000 students annually."

22. The requirements for documenting a "minimum of two" programs or projects for C4 and C6 involving "at least 200,000 students annually" are material because they are intended to

    provide the Department with assurance that the FCAT administration contractor can perform the large-scale, image-based scoring requirements of the contract from a technological perspective.

    Such experience would indicate that the bidder would have been required to address the sort of system issues described by Bramlett.

23. Dr. Heidorn testified that the number 200,000 was used in C4 and C6 "to indicate the level of magnitude of experience which represented for us a comfortable level to show that a contractor had enough experience to ultimately do the project that we were interested in completing." Dr. Fisher, who authored Appendix J, testified that the 200,000 figure was included in C4 and C6 because it was a number judged sufficiently characteristic of

    large-scale programs to be relevant for C4 and C6. Dr. Fisher further testified that the Department was interested in having information that a bidder’s experience included projects of a sufficient magnitude so that the bidder would have experienced the kinds of processing issues and concerns that arise in a large-scale testing program.

24. The Department emphasized this specific quantitative minimum requirement in response to a question raised at the Bidder’s Conference held on November 13, 1998:

    Q9: In Appendix J, the criteria for evaluating corporate quality for the administration operations C4, indicates that the bidder must have experience imaging as indicated. Does this mean that the bid [sic]

    must bid for using [sic] imaging technology for reading and mathematics tests?

    A: Yes. The writing assessment may be handscored for two years, and then it will be scored using imaging technology. To be responsive, a bid must be for imaging. The corporate experience required (200,000 students annually for which reports were produced in three months) could be the combined experience of the primary contractor and the subcontractors. (Emphasis added.)

25. Criterion C4 addresses the RFP work tasks relating to handscoring, including both the image-based handscoring of the reading and math portions of the FCAT for all administrations and the writing portions of the FCAT for later administrations. The "Work Task" column for C4 in Appendix J of the RFP states:

    Design and implement efficient and effective procedures for handscoring student responses to performance tasks within the limited time constraints of the assessment schedule.

    Handscoring involves image-based scoring of reading and mathematics tasks for all administrations and writing tasks for later administrations at secure scoring sites.

    Retrieve and score student responses from early district sample schools and deliver required data to the test development contractor within critical time periods for calibration and scaling.

26. The "Necessary Documentation" column for C4 in Appendix J states:

    Bidder must document successful completion of a minimum of two performance item scoring projects for statewide assessment programs during the last four years for which the bidder was required to perform as described in the Criteria column. (Emphasis added.)

    
    

27. The "Criteria" column for C4 in Appendix J, like the related work tasks in the RFP, addresses both image-based

    handscoring of reading and math, as well as paper-based or image- based handscoring of writing. In connection with all handscoring work tasks, "[t]he bidder must demonstrate completion of test administration projects for a statewide program for which performance items were scored using scoring rubrics and associated scoring protocols." With respect to the work tasks for handscoring the reading and math portions of the FCAT, "[t]he bidder must demonstrate completion of statewide assessment programs involving scoring multiple-choice and performance items for at least 200,000 students annually for which reports were produced in three months." In addition, for the reading and math work tasks, "[e]xperience must been shown in the use of imaging technology and hand-scoring student written responses with completion of scoring within limited time restrictions." This provision dealing with "imaging technology" experience self-evidently addresses the reading and math components, because separate language addresses imaging experience in connection with the writing component. The relevant handscoring experience for the reading and math aspects of the program is experience using image-based technology. By contrast, with respect to the work tasks for scoring the writing portions of the FCAT, "the bidder must also demonstrate completion of statewide assessment programs involving paper-based or imaged scoring student responses to writing assessment prompts for at least 200,000 students annually for which reports were produced in three months." (Emphasis added.)

28. Criterion C6 addresses work tasks relating to designing and implementing systems for processing, scanning, imaging and scoring student responses to mixed-format tests within limited time constraints. The "Work Task" column for C6 in RFP Appendix J states:

    Design and implement systems for the processing, scanning, imaging, and scoring of student responses to test forms incorporating both multiple-choice and constructed response items (mixed-format) within the limited time constraints of the assessment schedule.

    Scoring of student responses involves implementation of IRT scoring tables and software provided by the development contractor within critical time periods.

29. The "Necessary Documentation" column for C6 in Appendix J states:

    Bidder must document successful completion of a minimum of two test administration projects for statewide assessment programs during the last four years in which the bidder was required to perform as described in the Criteria column. (Emphasis added.)

    
    

30. The Criteria column for C6 in Appendix J states:

    
    

    The bidder must demonstrate completion of test administration projects for statewide assessment programs or other large-scale assessment programs that required the bidder to design and implement systems for processing, scanning, imaging, and scoring responses to mixed-format tests for at least 200,000 students annually for which reports were produced in three months. Experience must be shown in use of imaging student responses for online presentation to readers during handscoring. (Emphasis added.)

    RFP Provisions Per Corporate Qualifications

    
    

31. The procedure for evaluating a bidder’s corporate qualifications is described in RFP Section 11.3:

    The Department will evaluate how well the resources and experience described in each bidder’s proposal qualify the bidder to provide the services required by the provisions of this RFP. Consideration will be given to the length of time and the extent to which the bidder and any proposed subcontractors have been providing services similar or identical to those requested in this RFP. The bidder’s personnel resources as well as the bidder’s computer, financial, and other technological resources will be considered in evaluating a bidder’s qualifications to meet the requirements of this RFP. Client references will be contacted and such reference checks will be used in judging a bidder’s qualifications.

    
    

    The criteria to be used to rate a bidder’s corporate qualifications to meet the requirements of this RFP are shown in Appendix J and will be applied as follows:

    
    

    * * *

    
    

    Administrative Activities. Each of the nine administration activities criteria in Appendix J will be individually rated by members of the evaluation team. The team members will use the rating scale shown in Figure 1 below.

    Individual team members will review the bidder’s corporate qualifications and rate the response with a rating of one to five. The ratings across all evaluators for each factor will be averaged, rounded to the nearest tenth, and summed across all criteria. If each evaluator assigns the maximum number of points for each criterion, the total number of points will be 45. To meet the requirements of Stage II, the proposal must achieve a minimum rating of 27 points and have no individual criterion for which the number of

    points averaged across evaluators and then rounded is less than 3.0.

    
    

    Each proposal that receives a qualifying score based on the evaluation of the bidder’s qualifications will be further evaluated in Stage III.

    
    

    Figure 1

    Evaluation Scale for Corporate Qualifications

    5 Excellent	4	3 Satisfactory	2	1 Unsatisfactory
    The bidder has demonstrated exceptional experience and capability to perform the required tasks.		The bidder has demonstrated that it meets an acceptable level of experience and capability to perform the required tasks.		The bidder either has not established its corporate qualifications or does not have adequate qualifications.

    
    

32. RFP Section 11.3 provides that each of the nine corporate qualifications criteria for administration operations in Appendix J (C1 through C9) will be individually rated by the six members of the evaluation team using a scale of one to five. A rating of three is designated as "satisfactory" which means that "[t]he bidder has demonstrated that it meets an acceptable level of experience and capability to perform the required tasks." In order to be further evaluated, Section 11.3 provides that there must be no individual corporate qualifications criterion for which the bidder’s proposal receives a score less than 3.0 (average points across evaluators). Dr. Fisher, the primary author of Section 11.3

    of the RFP, referred to the 3.0 rating as the "cut score." (Emphasis added.)

33. The RFP’s clear and unambiguous terms thus establish the "minimum threshold" of experience that a bidder "must demonstrate" in its proposal for Criterion C1 through Criterion C9. The "minimum threshold" of experience that a bidder must demonstrate for each criterion is described in Appendix J of the RFP.

34. If a proposal failed to demonstrate that the bidder meets the minimum threshold of experience for a particular criterion in Appendix J, the bidder obviously would not have demonstrated "that it meets an acceptable level of experience and capability to perform the required tasks." Thus, in that setting, an evaluator was to have assigned the proposal a rating of less than "satisfactory," or less than three, for that criterion. (Emphasis added.)

35. The fact that a score less than "3" was expected for -- and would eliminate -- proposals that did not demonstrate the "minimum threshold" of experience does not render meaningless the potential scores of "1" and "2." Those scores may reflect the degree to which a bidder’s demonstrated experience was judged to fall below the threshold. Although some corporate capability minimums were stated quantitatively (i.e., "minimum of two," or "at least 200,000"), others were open to a more qualitative assessment (i.e., "large-scale," "systems," or "reports"). Moreover, a proposal that included demonstrated experience in some manner

    responsive to each aspect of Appendix J might nevertheless be assigned a score of less than "3," based on how an evaluator assessed the quality of the experience described in the proposal. By the terms of the RFP, however, an average score across evaluators of less than 3 represented essentially a decision that the minimum threshold of experience was not demonstrated.

36. Had the Department truly intended Appendix J to reflect only general targets or guidelines, there were many alternative ways to communicate such an intent without giving mandatory direction about what bidders "must demonstrate" or without establishing quantitative minimums (i.e. "a minimum of two," or "at least 200,000"). RFP Appendix K, for instance, sets forth the evaluation criteria for technical proposals in broad terms that do not require the bidder to provide anything in particular. Even within Appendix J, other than in Criterion C4 and Criterion C6, bidders were to show experience with "large-scale" projects rather than experience at a quantified level.

37. Pursuant to the RFP’s plain language, in order to meet the "minimum threshold" of experience for Criterion C4 and Criterion C6, a bidder "must demonstrate," among other things, successful completion of a "minimum of two" projects, each involving the use of image-based scoring technology in administering tests to "at least 200,000 students annually."

    Department’s Evaluation of Corporate Qualifications

    
    

38. In evaluating Harcourt’s proposal, the Department failed to give effect to the plain RFP language stating that a bidder "must document" successful completion of a "minimum of two" testing projects involving "at least 200,000 students annually" in order to meet the "minimum threshold" of experience for C4 and C6.

39. Dr. Fisher was the primary author of Sections 10, 11 and Appendix J of the RFP. He testified that during the Stage II evaluation of corporate qualifications, the evaluation team applied a "holistic" approach, like that used in grading open-ended written responses in student test assessments. Under the holistic approach that Dr. Fisher described, each member of the evaluation team was to study the proposals, compare the information in the proposals to everything contained in Appendix J, and then assign a rating for each criterion in Appendix J based on "how well" the evaluator felt the proposal meets the needs of the agency.

40. Notwithstanding Dr. Fisher’s present position, the RFP’s terms and their context demonstrate that the minimum requirements for corporate qualifications are in RFP Appendix J.

41. During the hearing, Dr. Fisher was twice asked to identify language in the RFP indicating that the Department would apply a "holistic" approach when evaluating corporate qualifications. Both times, Dr. Fisher was unable to point to any explicit RFP language putting bidders on notice that the Department would be using a "holistic" approach to evaluating proposals and

    treating the Appendix J thresholds merely as targets. In addition, Dr. Fisher testified that the Department did not engage in any discussion at the bidders’ conference about the evaluation method that was going to be used other than drawing the bidders’ attention to the language in the RFP.

42. As written, the RFP establishes minimum thresholds of experience to be demonstrated. Where, as in the RFP, certain of those minimum thresholds are spelled out in quantitative terms that are not open to interpretation or judgment, it is neither reasonable nor logical to rate a proposal as having demonstrated "an acceptable level of experience" when it has not demonstrated the specified minimum levels, even if other requirements with which it was grouped were satisfied.

43. The plain RFP language unambiguously indicates that an analytic method, not a "holistic" method, will be applied in evaluating corporate qualifications. Dr. Fisher acknowledged that, in an assessment using an analytic method, there is considerable effort placed up front in deciding the specific factors that will be analyzed and those factors are listed and explained. Dr. Fisher admitted that the Department went into considerable detail in Appendix J of the RFP to explain to the bidders the minimums they had to demonstrate and the documentation that was required. In addition, Dr. Orr, who served as a member of the evaluation team and who herself develops student assessment tests, stated that in assessments using the "holistic" method there is a scoring rubric

    applied, but that rubric does not contain minimum criteria like those found in the RFP for FCAT.

44. The holistic method applied by the Department ignores very specific RFP language which spells out minimum requirements for corporate qualifications.

    Harcourt’s Corporate Qualifications for C4 and C6

    
    

45. Harcourt’s proposal lists the same three projects administered by Harcourt for both Criterion C4 and Criterion C6: the Connecticut Mastery Test ("CMT"), the Connecticut Academic Performance Test ("CAPT") and the Delaware Student Testing Program ("DSTP"). Harcourt’s proposal also lists for Criterion C4 projects administered by its proposed scoring subcontractors, Measurement Incorporated ("MI") and Data Recognition Corporation ("DRC"). However, none of the projects listed for MI or DRC involve image- based scoring. Thus, the MI and DRC projects do not demonstrate any volume of image-based scoring as required by C6 and by the portion of C4 which relates to the work task for the imaged-based scoring of the math and reading portions of the FCAT.

46. Harcourt’s proposal states that "[a]pproximately 35,000 students per year in grade 10 are tested with the CAPT."

    Harcourt’s proposal states that "[a]pproximately 120,000 students per year in grades 4, 6 and 8 are tested with the CMT." Harcourt’s proposal states that "[a]pproximately 40,000 students in grades 3, 5, 8, and 10" are tested with the DSTP. Although the descriptions of the CMT and the CAPT in Harcourt’s proposal discuss image-based

    scoring, there is nothing in the description of the DSTP that addresses image-based scoring. There is no evidence that the evaluators were ever made aware that the DSTP involved image-based scoring. Moreover, although the Department called the Delaware Department of Education ("DDOE") as a reference for Harcourt’s development proposal, the Department did not discuss Harcourt’s administration of the DSTP (including whether the DSTP involves image-based scoring) with the DDOE.

47. Harcourt overstated the number of students tested in the projects it referenced to demonstrate experience with image-based scoring. Harcourt admitted at hearing that, prior to submitting its proposal, Harcourt had never tested 120,000 students with the CMT. In fact, the total number of students tested by Harcourt on an annual basis under the CMT has ranged from 110,273 in the 1996-

    97 school year to 116,679 in the 1998-99 school year. Harcourt also admitted at hearing that, prior to submitting its proposal, Harcourt had never tested 35,000 students in grade 10 with the CAPT. Instead, the total number of grade 10 students tested by Harcourt on an annual basis with the CAPT ranged from 30,243 in 1997 to 31,390 in 1998. In addition, Harcourt admitted at hearing that, prior to submitting its proposal, it had conducted only one "live" administration of the DSTP (as distinguished from field testing). That administration of the DSTP involved only 33,051, not 40,000, students in grades 3, 5, 8 and 10. Harcourt itself recognized that "field tests" of the DSTP are not responsive to C4

    and C6, as evidenced by Harcourt’s own decision not to include in its proposal the number of students field tested under the DSTP.

48. Even assuming that the numbers in Harcourt’s proposal are accurate, and that the description of the DSTP in Harcourt’s proposal reflected image-based scoring, Harcourt’s proposal on its face does not document any single project administered by Harcourt for C4 or C6 involving image-based testing of more than 120,000 students annually. When the projects are aggregated, the total number of students claimed as tested annually still does not reach the level of "at least 200,000;" it comes to only 195,000, and it reaches that level only once due to the single administration of the DSTP. Moreover, even if that 195,000 were considered "close enough" to the 200,000 level required, it was achieved only one time, while Appendix J plainly directs that there be a minimum of two times that testing at that level has been performed.

49. The situation worsens for Harcourt when using the true numbers of students tested under the CMT, CAPT, and DSTP, because Harcourt cannot document any single image-based scoring project it has administered involving testing more than 116,679 students annually. Moreover, when the true numbers of students tested are aggregated, the total rises only to 181,120 students tested annually on one occasion, and no more than 141,663 tested annually on any other occasion.

50. Despite this shortfall from the minimum threshold of experience, under the Department’s holistic approach the evaluators

    assigned Harcourt’s proposal four ratings of 3.0 and two ratings of

    
    

    4.0 for C4, for an average of 3.3 on C4; and five ratings of 3.0 and one rating of 4.0 for C6, for an average of 3.2 on C6.

51. Applying the plain language of the RFP in Sections 10 and

    
    

    11 and Appendix J, Harcourt did not demonstrate that it meets an acceptable level of experience and capability for C4 or C6, because Harcourt did not satisfy the minimum threshold for each criterion by demonstrating a minimum of two prior completed projects involving image-based scoring requiring testing of at least 200,000 students annually. Harcourt’s proposal should not have received any rating of 3.0 or higher on C4 or C6 and should have been disqualified from further evaluation due to failure to demonstrate the minimum experience that the Department required in order to be assured that Harcourt can successfully administer the FCAT program.

    NCS’s Compliance With RFP Requirements

    
    

52. Even though the NCS proposal did not meet all of the mandatory requirements, and despite the requirement of Section

    11.2 that the proposal be automatically disqualified under such circumstances, the Department waived NCS’s noncompliance as a minor irregularity.

53. The factors in C4 and C6 were set, minimal requirements with which NCS did not comply. For example, one of the two programs NCS submitted in response to Criteria C4 and C6 was the National Assessment of Educational Progress program ("NAEP").

    NAEP, however, is not a "statewide assessment program" within the meaning of that term as used in Criteria C4 and C6.

54. Indeed, NCS admitted that NAEP is not a statewide assessment program and that, without consideration of that program, NCS’s proposal is not responsive to Criteria C4 and C6 because NCS has not have submitted the required proof of having administered two statewide assessment programs. This error cannot be cured by relying on the additional experience of NCS’s subcontractor because that experience does not show that its subcontractor produced reports within three months, and so such experience does not demonstrate compliance with Criteria C4.

55. The Department deliberately limited the competition for the FCAT contract to firms with specified minimum levels of experience. As opined at final hearing, if the Department in the RFP had announced to potential bidders that the type of experience it asked vendors to describe were only targets, goals and guidelines, and that a failure to demonstrate target levels of experience would not be disqualifying, then the competitive environment for this procurement would have differed since only

    2.06 evaluation points (out of a possible 150) separated the NCS and Harcourt scores. Dr. Heidorn conceded that multiple companies with experience in different aspects of the FCAT program -- a computer/imaging company and a firm experienced in educational testing -- might combine to perform a contract like the FCAT. Yet, that combination of firms would be discouraged

    from bidding because they could not demonstrate the minimum experience spelled out in the RFP. Language in the RFP, indicating the "holistic" evaluation that was to be applied, could have resulted in a different field of potential and actual bidders.

    CONCLUSIONS OF LAW

    
    

56. Protests arising from a contract bidding process are governed by Section 120.57(3) Florida Statutes, which provides in relevant part as follows:

    (f) [T]he burden of proof shall rest with the party protesting the proposed agency action. In a competitive-procurement protest, other than a rejection of all bids, the administrative law judge shall conduct a de novo proceeding to determine whether the agency’s proposed action is contrary to the agency’s governing statutes, the agency’s rules or policies, or the bid or proposal specifications. The standard of proof for such proceedings shall be whether the proposed agency action was clearly erroneous, contrary to competition, arbitrary, or capricious.

57. The issue in the instant case is whether the agency’s proposed action is contrary to the "bid or proposal specifications." No protest has been raised that the proposed action is violative of governing statutes of the agency, or its rules or policies. Accordingly, no conclusions are reached in that regard.

58. NCS has shown that Harcourt’s bid was non-responsive due to its failure to meet the minimum threshold of experience for corporate qualifications for C4 and C6. Notably, NCS’s proposal

    also fails to satisfy the minimum requirements for corporate qualifications for C4 and C6.

59. Section 287.057(2), Florida Statutes, states, in part, that if an agency chooses to procure commodities or contractual services through a request for proposals, it must include:

    1. statement of the commodities or contractual services sought and all contractual terms and conditions applicable to the procurement of commodities or contractual services, including the criteria, which shall include, but need not be limited to, price, to be used in determining acceptability of the proposal. . . .

       
       

60. Section 287.057(2), Florida Statutes, also provides:

    
    

    The award shall be made to the responsible offeror whose proposal is determined in writing to be the most advantageous to the state, taking into consideration the price and the other evaluation criteria set forth in the request for proposals . . . .

    
    

61. Further, Section 287.012(15), Florida Statutes, states, in part:

    Requests for proposals shall state the relative importance of price and any other evaluation criteria.

    
    

62. The terms of the RFP relating to what a bidder must include in its proposal with respect to corporate qualifications are clear and specific. The RFP requires a bidder to demonstrate in its proposal that it has achieved the minimum threshold of experience for C4 and C6 as set forth in Appendix J of the RFP in order for its bid to be acceptable during Stage II of the

    evaluation process. For C4 and C6, Appendix J states that the bidder "must document" a "minimum of two" projects using image- based scoring that involve "at least 200,000 students annually." These minimum requirements for corporate qualifications in the RFP are mandatory as evidenced by the use of the word "must." See Jacobs Assoc., Inc. v. Department of Corrections, DOAH Case No. 96- 5831BID (Recommended Order March 4, 1997) (Final Order April 3, 1997) ("[c]ertain of the provisions were deemed mandatory requirements, generally by the use of shall, must or will . . .

    ."). Moreover, the minimum requirements for corporate qualifications in the RFP are material because they provide the Department with some level of assurance that the administration contractor will be able to successfully perform a project of the magnitude of the FCAT, particularly with respect to the technology required to successfully administer a large-scale image-based scoring project. See Robinson Elec. Co., Inc. v. Dade County, 417 So. 2d 1032, 1034 (Fla. 3d DCA 1982).

63. The minimum requirements for corporate qualifications for C4 and C6 in the RFP were ignored by the Department in conducting what it has referred to as the "holistic" method of evaluation in Stage II of the RFP evaluation process. While the "holistic" method of evaluation may be appropriate in certain test assessments, it was not appropriate in this procurement which clearly and specifically defined minimum criteria for corporate qualifications. By ignoring material requirements of the RFP, the

    Department improperly failed to use prescribed criteria there by engaging in a fatally defective decision not based on an honest and reasonable exercise of agency discretion. See Marpan Supply Co., Inc. v. Department of Management Servs., DOAH Case No. 96-2777BID (Recommended Order Sept. 26, 1996) (Final Order Nov. 22, 1996) (citing Courtenay v. Department of Health and Rehabilitative Servs., 581 So. 2d 621, 623 (Fla. 5th DCA 1991)).

64. The Department should reject all bids, and re-bid the administration component of the FCAT procurement.

RECOMMENDATION

Based on the foregoing Findings of Fact and Conclusions of Law, it is recommended that Respondent, State of Florida, Department of Education, enter a Final Order rejecting the bids submitted by Harcourt and NCS for the administration component of the RFP. The Department should then seek new proposals.

DONE AND ENTERED this 25th day of May, 1999, in Tallahassee, Leon County, Florida.

DON W. DAVIS

Administrative Law Judge

Division of Administrative Hearings The DeSoto Building

1230 Apalachee Parkway

Tallahassee, Florida 32399-3060

(850) 488-9675 SUNCOM 278-9675

Fax Filing (850) 921-6847 www.doah.state.fl.us

Filed with the Clerk of the Division of Administrative Hearings this 25th day of May, 1999.

COPIES FURNISHED:

Karen D. Walker, Esquire Holland and Knight, LLP Post Office Drawer 810 Tallahassee, Florida 32302

Mark D. Colley, Esquire Holland and Knight, LLP Suite 400

2100 Pennsylvania Avenue, Northwest Washington, D.C. 20037

Charles S. Ruberg, Esquire Department of Education The Capitol, Suite 1701

Tallahassee, Florida 32399-0400

Paul R. Ezatoff, Jr., Esquire Christopher B. Lunny, Esquire Katz, Kutter, Haigler, Alderman,

Bryant and Yon, P.A.

106 East College Avenue, Suite 1200 Tallahassee, Florida 32302-7741

Tom Gallagher Commissioner of Education Department of Education

The Capitol, Plaza Level 08 Tallahassee, Florida 32399-0400

Michael H. Olenick, General Counsel Department of Education

The Capitol, Suite 1701 Tallahassee, Florida 32399-0400

NOTICE OF RIGHT TO SUBMIT EXCEPTIONS

All parties have the right to submit written exceptions within 10 days from the date of this Recommended Order. Any exceptions to this Recommended Order should be filed with the agency that will issue the Final Order in this case.

Docket for Case No: 99-001226BID

Orders for Case No: 99-001226BID