Findings Of Fact Based upon all of the evidence, the following findings of fact are determined: Background On August 16, 1991, respondent, Department of Professional Regulation (DPR), issued Request for Proposal No. 92-002 (RFP) to various firms inviting them to submit proposals for assisting DPR and the Florida Real Estate Commission (Commission) in the production and scoring of the Florida Real Estate and Appraisal Examination for the period beginning January 1, 1992, through June 30, 1993, with a renewal option for one to two years. More specifically, the contract called for the successful firm to develop examinations from a bank of questions provided by the Commission, administer periodic examinations, score and analyze tests, and perform other related support services. Such proposals were to filed no later than 2:00 p.m. on September 20, 1991. Thereafter, and in accordance with the RFP, a six-person committee composed of representatives of the DPR, Commission and Division of Real Estate (Division) would evaluate the proposals and make a recommendation to the Secretary of DPR, who retained final authority to award the contract. Proposals were timely filed by three firms, including petitioner, Professional Testing Service, Inc. (PTS), Applied Measurement Professionals, Inc. (AMP), and National Assessment Institute (NAI). After these proposals were evaluated by the committee, AMP was recommended for award of the contract. By a 4-3 vote, the Commission concurred with this recommendation. On October 24, 1991, the Secretary of DPR selected AMP as the recipient of the contract, and notice to that effect was posted. Citing various alleged irregularities in the review process, PTS filed its formal protest to the award of the contract on November 4, 1991. After efforts to informally resolve this dispute were unsuccessful, this proceeding ensued. The Vendors Who Submitted Proposals Petitioner is a Florida corporation with offices located at 223 Pasadena Place, Orlando, Florida. It has provided various licensure examination services to DPR for the past eighteen years. Except for a two year break in 1984 and 1985, for the last eleven years PTS has held the contract with DPR to provide licensure examination services on the real estate examination, and pending resolution of this dispute, it continues to provide such services. AMP did not intervene as a party in this cause. However, according to its proposal, AMP is a Kansas corporation with offices at 8310 Nieman Road, Lenexa, Kansas, and has been in the business of developing and administering examinations since 1984. Among other things, it now provides assistance to the State of Michigan for the real estate licensure examination. AMP maintains no offices or personnel in the State of Florida. Although NAI submitted a response to the RFP, it is not a party to this action. Located in Clearwater, Florida, with branch offices in other cities in Florida and throughout the country, NAI has provided assessment services to various state governments since 1976. It has administered the DPR contractor examination for the last fourteen years. Events Leading to the Issuance of RFP 92-002 In May 1991, DPR issued RFP 91-009 requesting proposals for assistance in the development and administering of real estate examinations. That RFP requested proposals on three components of the real estate examination services, including (a) application processing, (b) test development, grading and review, and (c) administration of the examination. Items (a) and (c) and a portion of item (b) had previously been handled by the Division, the administrative arm of the Commission. In response to RFP 91-009, proposals were filed by PTS, AMP, NAI, and two other vendors. PTS was recommended for award of the contract as to two components while AMP was recommended for the award of the contract as to one component even though its proposal had been deemed to be nonresponsive. However, DPR eventually decided to reject all proposals and issue a new RFP. The new RFP (91-002) did not call for proposals on items (a) and (c) and the test development portion of item (b) since the Division determined that it would continue to perform those functions. Even so, while RFP 91-002 was more narrow in scope of services than was 91-009, it nonetheless contained some of the same terms, conditions and requirements as did the earlier RFP. It is also noted that, with one minor exception relating to on-site grading, the scope of services under the existing contract held by PTS and those enumerated in RFP 91- 002 are the same. Prior to the time for submitting responses to RFP 91-009, the Division, whose director was designated as contract manager, and Commission conducted a pre-bid conference with interested vendors to answer questions concerning the RFP. Both RFP 91-009 and 91-002 contained an identical requirement that the successful firm "(m)aintain an office/center in the greater Orlando area." In response to a question by a vendor as to whether DPR would "accept proposals that identify Tallahassee as the office/center site for the provider's office in lieu of Orlando", the response given was "no, Orlando is the designated site." Other questions and answers concerning the same topic were as follows: Q. Would the Department consider allowing the contractor (for application processing function only) to maintain offices in Tallahassee instead of Orlando, provided that 1)regular meetings are held in Orlando, and 2) reports can be delivered to the Department's offices in Orlando? A. No. Because the application processing function relating to deficient applications must be closely coordinated with the records section, the investigative section, the legal section, administrative section of the Division of Real Estate and the Florida Real Estate Commission. Florida Statutes require that the office of the Division of Real Estate and the Florida Real Estate Commission be located in Orlando. Therefore, Divisional functions must be performed in the Orlando area. Q. Is Orlando the only site the Department will agree to, or is the Department flexible to the location if all requirements can be met from another site in the State of Florida? Orlando is the only site. The above questions and answers, together with others asked and answered at the conference, were transcribed by DPR and then furnished to all interested vendors. Since AMP filed a proposal in response to RFP 91-009, it may be reasonably inferred that AMP was privy to the above clarifying information. It is noted that there was no pre-bid conference prior to the filing of proposals in response to RFP 91-002. The RFP A copy of RFP 91-002 is found on pages 90 through 108 of petitioner's composite exhibit 1. The RFP includes sections relating to statement of need, purpose, scope of work, DPR responsibilities, provider responsibilities, general information, documents required in submitting a proposal, the proposal format, proposal rating criteria, and an appendix delineating the manner in which points would be awarded in the evaluation process. Prior to the submission of the proposals, no vendor formally challenged any provision in the RFP as being unreasonable, ambiguous, or otherwise unlawful in any respect. A number of provisions within RFP 91-002 are relevant to this controversy and are cited below. First, there is a general requirement found in Article VIII which requires all responses to be prepared in a manner consistent with the requirements of the RFP. More specifically, this article provided that Respondents must follow the proposal format as set forth under Section III in this request for proposal. The provider shall refer to the request for proposal to ensure all required information is submitted. By following the designed format of proposals, respondents will have a uniform method of presenting information helping assure complete fairness by the review team in evaluating proposals. Next, Article VII set forth those documents that were required to be submitted with the proposal. Among them was a requirement in paragraph 1 that the vendor submit "evidence that the organization is a legal entity." The purpose of this requirement was to ensure that DPR could enforce the agreement in the event the successful firm later attempted to back out of its commitment. The RFP also prescribed twenty-three major services that were within the scope of work to be provided by the successful firm. Indeed, the RFP stated in unequivocal language that "(t)he provider shall perform the following services as outlined below." (Emphasis added) Specifically, paragraphs 22 and 23 of Article III specified that the successful firm would Furnish to each candidate an unofficial grade report at the examination site at no additional cost to the candidate according to specifications approved by the Department. Maintain an office/center in the greater Orlando area. These two requirements, both unambiguous, were reasonably construed by participating vendors to mean that (a) the successful firm would furnish, at no cost to the candidate, on-site grade reports to those candidates who desired an unofficial grade after the examination was completed, and (b) the successful firm would maintain an office in the Orlando area to provide technical and other assistance to Commission and Division personnel regarding the examination. The interpretation as to paragraph 23 was especially reasonable since it conformed to clarifying information given to vendors at the pre-bid conference prior to the submission of proposals for RFP 91-009. Finally, despite DPR's contention to the contrary at hearing, the requirements in paragraphs 22 and 23 were considered material by the Division and Commission. From DPR's perspective, an important consideration was the costs to be charged both the candidate who sat for the examination and the candidate who made application but did not take the examination. This was because DPR's experience indicated that each year a large number of candidates apply for the examination but then fail to appear and sit for the examination. Indeed, during the most recent fiscal year of record, DPR had received 38,886 applications to take the various real estate and appraiser examinations but almost 3,000 did not appear. These figures were contained in RFP 91-009 and thus were available to all vendors, including AMP. Accordingly, DPR inserted a provision in the RFP directing each vendor to develop a cost schedule reflecting the cost for both the candidates scheduled for examination and the candidates who were actually examined. This material requirement was embodied in paragraph 1 of Article VI, which provided the following instructions to the vendors: The costs schedule for this proposal shall be priced on a per candidate examined for the first year and each of the option years provided. *Differences between the number of candidates scheduled for exam and the number of candidates examined shall be compensated for at a specific rate per candidate to be set forth by the provider.* (Emphasis added between *) The above provision was consistent with the manner in which the existing contract holder (PTS) had calculated its candidate costs for DPR during the preceding five years. Very simply, this meant that the proposal had to include one cost figure for candidates examined and another cost figure for scheduled candidates who did not appear. In calculating the costs for scheduled candidates, Appendix I, Section IV, page 3 of 3, required that all vendors develop a cost for services, including a "cost per candidate scheduled" to be made up of eight cost components: scan sheet costs, examination booklet production costs, scanning and microfilming costs, on-site grading costs, scoring costs, grade reporting/grade summary costs, security costs, and item bank maintenance costs. Therefore, each vendor was required to segregrate its costs per candidate into the eight prescribed categories, with the sum of those eight components representing the total costs per candidate scheduled. The Responses A copy of AMP's proposal filed on September 20, 1991, is found at pages 257 through 392 of petitioner's exhibit 1. In response to the requirement that the vendor give "evidence that the organization is a legal entity", AMP responded that it was a private stock corporation incorporated in 1982 in the State of Kansas. Although AMP represented that "a Certificate of Good Standing with the State of Kansas is available upon request," no such certificate was enclosed with its proposal. Documentation offered by PTS confirmed that AMP is not a Florida corporation, and there is no evidence to show that AMP, as an out- of-state corporation, has registered with the Department of State to transact business in the State of Florida. In its proposal, AMP provided an overall price "per candidate" but failed to differentiate between the costs incurred for candidates examined and candidates who were scheduled to take the examination but did not appear. This was contrary to the requirement in Article VI that such costs be identified for both categories and caused the proposal to be nonresponsive in a material respect. Although the RFP specifically required the vendor to set out eight cost components in developing the cost per candidate scheduled, AMP submitted nine specific costs as follows: 1. Test Development $3.72 2. Scan Sheet 0.10 3. Examination Booklet Production 2.47 4. Scanning 1.01 5. Scoring 1.05 6. Grade Tape Preparation 0.43 7. Security 0.32 8. Item Bank Maintenance 0.50 9. On-Site Grading 0.20 TOTAL $9.80 It should be noted that the first item, "Test Development" costs, which included 38 percent of AMP's total price, was not a category contained in or authorized by the RFP. Thus, the proposal was nonresponsive in this material respect. According to its proposal, AMP did not intend to maintain an office/center in the greater Orlando area. Rather, it proposed that: as an alternative to staffing an office in Orlando, AMP proposes to conduct regular monthly meetings in Orlando with the Division and key AMP project staff, typically the doctorate level Program Director and a Test Development Specialist. AMP will bring its portable computer equipment, and paper and computer files of the draft test and the item banks to these meetings. Any changes to examinations, as required by the Division, can then be immediately made in the draft tests and reviewed by the Division. It is AMP's opinion that this procedure will provide the Division with the direct responsiveness desired, and ensure efficient communication between the Division and the key project staff, without information being filtered through a lesser qualified individual in a satellite office. Additional urgent concerns can be discussed using the telephone or FAX equipment with the Program Director. (page 272, petitioner's exhibit 1) By filing this response, AMP contravened the material requirement in paragraph 23 of Article III that it maintain an office/center in the greater Orlando area. In response to the material requirement that it "furnish to each candidate an unofficial grade report at the examination site at no additional cost to the candidate according to specifications approved by the Department", AMP responded that it would: train the Department's test administration personnel to use this equipment to this end . . . . AMP's cost proposal is based on using the Department's personnel. If this is not possible, an additional per candidate fee will be determined based on using AMP personnel. Thus, AMP's response unilaterally modified the RFP requirement that AMP personnel rather than DPR personnel perform the task of providing unofficial grade reports at the examination site. Moreover, AMP's proposal did not state what its price per candidate would be if DPR enforced the RFP requirement. Under AMP's proposal, it calculated a cost of 20 cents per candidate for providing this service assuming DPR personnel were used. This was $2.27 less per candidate than the charge given by PTS ($2.47). Thus, AMP gained an economic advantage by its failure to provide on-site grading with its own personnel. Accordingly, this portion of the proposal was nonresponsive. The proposal of PTS was responsive to the RFP in all respects. This finding was not contradicted by DPR. It is noteworthy that two potential vendors, H. H. Block & Associates, Inc., a Gainesville, Florida firm, and Psychological Services, Inc., a firm located in Glendale, California, both advised DPR in writing that because of the requirement in the RFP that the successful vendor locate a center/office in the greater Orlando area, they would not be filing a proposal. The latter two vendors were obviously prejudiced by DPR's failure to advise them that it did not intend to enforce the requirement in paragraph 23 of Article III. This failure by DPR to enforce the provision also gave AMP an economic advantage over other vendors since AMP did not have to incur the costs of operating an Orlando office. The Evaluation Process The committee selected to evaluate the proposals was made up of six individuals appointed by the Secretary of DPR. It met in Tallahassee on September 30, 1991, to evaluate the proposals. The committee was chaired by Ella D. Hall, a DPR psychometrician. In accordance with the instructions in the RFP, the committee reviewed only the technical aspects of the proposals and did not review the cost data. Article IX of the RFP outlined the responsibilities of the evaluation committee and the procedure for evaluating proposals. Among other things, the committee was assigned the responsibility to: first determine if all required documents are included, that the proposal format is followed, and that all responses to the request's responsibilities of the provider are properly addressed. The cost data will not be reviewed as part of the technical evaluation. The committee utilized both a scoring guide and an evaluation guide in arriving at its recommendation. A total of 300 possible points were to be given to a vendor, of which 226 were related to technical matters. As noted above, the committee did not consider cost in its evaluation. Contrary to Article IX of the RFP, the committee did not initially determine whether the proposals were responsive. More specifically, the committee did not "first determine if all required documents (were) included, that the proposal format (was) followed, and that all responses to the request's responsibilities of the provider (were) properly addressed." According to the committee's chairperson, the committee was never told to determine if the proposals were responsive before evaluating them on their merits. Indeed, the chairperson assumed, albeit incorrectly, that someone else had previously evaluated the proposals in terms of responsiveness. As it turned out, none of the proposals were evaluated for responsiveness prior to or during the committee evaluation process. Through its chairperson, the committee issued a written report on October 9, 1991, recommending that AMP be awarded the contract. Although the committee considered the responses by AMP and PTS to be almost equal in terms of technical ratings, it gave a slight edge to AMP's proposal in the cost rating and recommended that AMP be awarded the contract. NAI was a very distant third in the evaluation process. Because of the committee's report, and AMP's announcement on October 16, 1991, as discussed below, that its price per candidate scheduled but not examined would be zero, the Commission voted 4-3 to endorse the committee's recommendation. The Secretary received the proposals in that posture. Events Occurring After the Committee Evaluation On October 4, 1991, or before the contract was awarded but after the committee evaluation was completed, AMP's president, Steven K. Bryant, sent a letter to the Secretary of DPR. The letter was received by DPR on October 9, 1991, and a copy of same has been received in evidence as petitioner's exhibit It provides in relevant part as follows: Dear Secretary Stuart: At the request of Lou Ritter, I am writing to you due to our concerns about the opportunity for our company to fairly compete for testing services business in your state. The recent rebidding of the Florida Cosmetology and Real Estate Programs have generated some serious questions in my mind as to whether or not it is in our interest to continue to respond to Florida RFPs. I hope you will investigate the following concerns and use the power of your office to correct these circumstances: The second RFP regarding real estate was clearly written to ensure that the current vendor retains the contract. The requirement that the vendor have an office in Orlando staffed by an individual whose only job would be to serve as a liaison with the Real Estate Commission makes it basically impossible for any organization based outside of the state of Florida to compete with the current vendor. An organization such as ours cannot afford to put a doctorate level individual in an office in Florida to be at the beck and call of the Real Estate Commission to answer examination development questions and economically survive. At best, we could put a clerical person in an office in Orlando who would be unable to answer any of the Commission's test development concerns. In our response to the RFP, we indicated that we would not establish an Orlando office, but would provide a doctoral level measurement expert and a test development specialist to meet once a month with the Real Estate Commission to revise the real estate examinations to their specifications using very highly qualified individuals. The second real estate RFP also requires that the examination booklets be printed and shipped to Florida examination centers within a seven day time frame, after the examinations are reviewed by the Real Estate Commission. This item was clearly written for the current vendor, which is basically photocopying examination materials as soon as the Commission provides approval on examination copy. There is no real reason why the examinations could not be reviewed and approved by the state to allow a longer time frame for printing, so that higher quality offset printed examination booklets could be prepared and shipped to Florida by a vendor residing out of state. In fact, we could provide several months of examinations in advance for the Commission's approval and avoid the crisis mode of printing which a seven day turnaround would cause. Although, we could provide the seven day turnaround through the use of overnight air carriers and the like, clearly the Real Estate Commission had in mind the current vendor when making this unreasonable requirement. * * * The letter also carries a handwritten note by the Secretary in the upper right hand corner which reads "Discussed with Lou Ritter/Steve Bryant - file 10/21". According to the Secretary, Ritter (a former Secretary of DPR) is now a consultant for AMP. The Secretary acknowledged that he spoke with Ritter and Bryant concerning the letter and advised them the agency was proceeding with the RFP. The Secretary's assertion that he followed up on the letter only to the extent that he wanted to ascertain if AMP had been unfairly penalized in the evaluation process was not contradicted. On October 16, 1991, Bryant sent a second letter on behalf of AMP to the Secretary of DPR. It read as follows: Dear Secretary Stuart: This is to confirm our telephone discussion regarding our price proposal for RFP #92-002 (sic) for Real Estate Examination Services. Since the RFP calls for AMP to provide examination booklets to the test centers the department administers, it was our intention to charge the state $9.80 per candidate tested, based on the number of answer sheets actually scored by AMP. Thus, there would not be a charge for candidates who do not appear for testing. Thank you for the opportunity to provide this information. Please let me know if there are any other questions. The above letter was solicted by the Secretary after he and Bryant spoke by telephone on or about October 15, 1991. The purpose of the letter was to allow AMP to clarify and amend its proposal which failed to include a cost for candidates who were scheduled to take the examination but did not appear. In contrast, neither PTS nor NAI were offered the opportunity to clarify or change their proposals after being filed. Thus, AMP was allowed to correct a material deviation from the terms of the RFP thereby giving it an advantage over its competitors. The contract was thereafter awarded to AMP on or about October 24, 1991. Summary of Errors in the RFP Process By failing to file evidence that it was a "legal entity", failing to file a cost per candidate scheduled but not examined, and submitting a price per candidate based on nine cost components rather than the prescribed eight, AMP failed to follow the proposal format required of all vendors by Article VIII. AMP's proposal was materially nonresponsive in four respects. First, it failed to differentiate between costs incurred for candidates examined and candidates who were scheduled but did not appear. Second, it utilized nine cost components in developing the cost per candidate scheduled instead of the eight components specified by the RFP. Third, in declining to establish an office in the greater Orlando area, AMP deviated from a material requirement. Fourth, contrary to the RFP, AMP proposed that DPR personnel rather than its own personnel provide unofficial grade reports to candidates at the examination sites. If this was unsatisfactory to DPR, AMP proposed to assess DPR an unspecified charge for providing this service. By waiving the enforcement of a material requirement (paragraph 23, Article III) after the proposals had been filed and evaluated, DPR gave a competitive advantage to AMP not enjoyed by other vendors. Further, by allowing AMP to correct a material variance from the RFP on October 16, 1991, as to the price charged for candidates scheduled but who did not take the examination, DPR gave a competitive advantage to AMP not enjoyed by other vendors. Collectively, these considerations support a finding that (a) the proposal submitted by AMP was materially nonresponsive and should be rejected, and (b) DPR created unfair competition and favoritism by waiving material requirements and allowing AMP to amend its proposal after being filed and evaluated.
Recommendation Based on the foregoing findings of facts and conclusions of law, it is, RECOMMENDED that a final order be entered by respondent rejecting the proposal filed by Applied Measurement Professionals, Inc. in response to RFP 91- 002 and awarding the contract to one of the other vendors who filed a response. DONE and ENTERED this 3rd day of January, 1992, in Tallahassee, Florida. DONALD R. ALEXANDER Hearing Officer Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-1550 (904) 488-9675 Filed with the Clerk of the Division of Administrative Hearings this 3rd day of January, 1992. APPENDIX Petitioner: Partially adopted in finding of fact 1. Partially adopted in findings of fact 11, 12, 13 and 22. Partially adopted in findings of fact 6 and 7. Partially adopted in finding of fact 7. Partially adopted in findings of fact 2, 3 and 4. Partially adopted in finding of fact 24. Partially adopted in findings of fact 10 and 14. Partially adopted in findings of fact 7, 11, 17, 19 and 29. Partially adopted in findings of fact 11, 18 and 29. Partially adopted in findings of fact 12, 15, 28 and 29. Partially adopted in findings of fact 13, 16, 28 and 29. Respondent:* 1. Partially adopted in finding of fact 1. 2-3. Partially adopted in finding of fact 22. Partially adopted in findings of fact 1, 20 and 24. Partially adopted in finding of fact 24. Partially adopted in finding of fact 13. 6a. Partially adopted in findings of fact 3, 15 and 19. 6b. Rejected as being contrary to the evidence. 6c. Partially adopted in finding of fact 16. 7-8. Partially adopted in finding of fact 24. 9-10. Partially adopted in finding of fact 26. Partially adopted in findings of fact 10 and 14. Partially adopted in finding of fact 16. Partially adopted in finding of fact 17. The second sentence is specifically rejected as being contrary to the evidence. Partially adopted in finding of fact 16. Partially adopted in finding of fact 11. Partially adopted in finding of fact 18. Rejected as being unnecessary. * Respondent's proposed findings of fact included numbers 4, 5, 6, 4, 5, 6. Therefore, the second set of findings numbered 4, 5 and 6 have been renumbered 6a, 6b and 6c for purposes of this Appendix. Note - Where a proposed finding has been partially adopted, the remainder has been rejected as being irrelevant, unnecessary, subordinate, not supported by the evidence, redundant, or a conclusion of law. COPIES FURNISHED: Joseph W. Lawrence, II, Esquire Post Office Box 1116 Fort Lauderdale, FL 33302 Vytas J. Urba, Esquire 1940 North Monroe Street, Suite 60 Tallahassee, FL 32399-0792 George Stuart, Secretary Department of Professional Regulation 1940 North Monroe Street Tallahassee, FL 32399-0750 Jack L. McRay, Esquire 1940 North Monroe Street, Suite 60 Tallahassee, FL 32399-0792
Findings Of Fact Donald A. Moyant, Petitioner, operates a contractors' exam school in Brandon, Florida, at which he preps builders to take the CILB exam for licensure as contractors. Petitioner took the exam given in June 1985 and received a score of 58.60. A score of 70.00 is required to pass this examination. The only answer that Petitioner questions is the answer he supplied to question No. 2, which related to the number of reinforcing bars required in a concrete slab of given dimensions and spacings of the bars. In Petitioner's solution to the problem he dropped the fractional bar that resulted from dividing the length of the slab by the spacing between bars. The correct solution requires the addition of a bar for this fractional spare. Petitioner's purpose in pursuing this proceeding is to obtain information to use in his Contractors Exam School, Inc. Since he would not have passed the examination even if his solution to question No. 2 had been marked correct, Petitioner is without standing to bring this action because he is doing so on behalf of the school he runs and not to allow him to receive a passing grade.
Findings Of Fact Based on the stipulations and admissions of the parties, the exhibits received in evidence, and the testimony of the witnesses at the hearing in this case, I make the following findings of fact. Findings based on stipulations and admissions of the parties The Petitioners, John Eugene Harden and Dova Cauthen, qualified for and were administered the January 1982 Electrical Contractors' Licensing Examination. After their examinations were graded and regraded, Petitioners were notified that they had not received a "Passing" score of 75 or more. The Petitioners, John Eugene Harden and Dova Cauthen, qualified for and were administered the July 1982 Electrical Contractors' Licensing Examination. After their examinations were graded and regraded, Petitioners were notified that they had not received a "Passing" score of 75 or more. Both Petitioners reside in Dade County, Florida. The Respondent Department of Professional Regulation (hereafter DPR or the "Department") is an agency of the State of Florida. The Respondent Electrical Contractors' Licensing Board (hereafter referred to as the "Board") is an agency of the State of Florida statutorily responsible, along with DPR, for licensing certified electrical contractors. The Office of the Executive Director of the Board is located at 130 North Monroe Street, Tallahassee, Florida. The business address of Petitioner Harden is Harden Electric, 311 N.E. 8th Street, Homestead, Florida 33030. The business address of Petitioner Cauthen is 959 N.E. 79th Street, Miami, Florida 33138. 8. On March 20, 1981, Rules 21GG-6.01 (2), 21GG-6.01 (3), 21GG-6.01 (4), of the Rules of the Electrical Contractors' Licensing Board, were amended. Rule 21GG-6.01 (2) altered the format of the licensing examination so that the formerly entirely open book examination became a part open book, part closed book examination. Rule 21GG-6.01 (4) raised the passing grade on the licensing examination from 70 to 75. Rule 21GG-6.01 (3) altered the technical format. On December 3, 1981, Susan Tully, counsel to the Electrical Contractors' Licensing Board, caused a change to be made in Rule 21GG-6.01 (1) (c) of the Rules of the Electrical Contractors' Licensing Board. This change was made without formal notice or informal notice to the public and without a vote of the Electrical Contractors' Licensing Board, although the change was discussed at a Board meeting. None of the procedures set forth in Section 120.54 of the Florida Statutes were followed in amending this Rule. Liz Cloud was the Bureau Chief of the Division of Elections, Bureau of Administrative Code, Department of State. The address of the Bureau is Room 1802, The Capitol Building, Tallahassee, Florida. The change in Rule 21GG-6.01 (1)(c) eliminated parts 72 A, B, C and D and indicated that the entire Fire Safety Code (and not just the aforementioned parts) would be a subject of the Electrical Contractors' Licensing Examinations. In fact, the reason for the alleged "technical change" was that the Board intended and tested examinees in the January and July 1982 Licensing Examinations on materials in parts of the Fire Safety Code in addition to those contained in 72 A, B, C and D of the Fire Safety Code. Petitioner Harden specifically requested to review his January and July 1982 Electrical Contractors' Licensing Examinations. On March 25, 1982, Petitioner Harden went to Tallahassee, Florida to review his January 1982 Licensing Examination papers, but although he requested an "examination review" he was never provided with his own examination booklet or a copy thereof. During this alleged "examination review" Mr. Harden wrote written objections to the January 1982 examination on the forms provided to him based on the master copy of the examination provided to him. On August 31, 1982, Petitioner Harden went to Tallahassee, Florida to review his July 1982 Licensing Examination papers, but although he requested an "examination review" he was never provided with his own examination booklet or a copy thereof. During this alleged "examination review" Mr. Harden wrote written objections to the July 1982 examination on the forms provided to him based on the master copy of the examination provided to him. Petitioner Cauthen specifically requested an "examination review" with respect to her January and July 1982 Licensing Examination papers. On March 17, 1982, Petitioner Cauthen went to Tallahassee, Florida to review her January 1982 Licensing Exami-nation papers, but although she requested an "examination review" she was never provided with her own examination booklet or a copy thereof. During this alleged "examination review" Ms. Cauthen wrote written objections to the January 1982 examination on the forms provided to her based on the master copy of the examination provided to her. On August, 19, 1982 Petitioner Cauthen went to Tallahassee, Florida to review her July 1982 Licensing Examination papers, but although she requested an "examination review" she was never provided with her own examination booklet or a copy thereof. During this alleged "examination review" Ms. Cauthen wrote written objections to the July 1982 examination on the forms provided to her based on the master copy of the examination provided to her. Petitioners Harden and Cauthen sought Board review of their January and July 1982 Electrical Contractors' Licensing Examinations. Petitioners were advised in July 1983 that no copies of their actual booklets exist; Petitioners were advised in October 1983 that their actual booklets were shredded. At the November 8, 1982, meeting of the Board, Ms. Ida Cameron representing DPR, presented to the Board a package of information (assembled in package form) for the Board to consider during the examination review. With regard to Question Number 71 on the afternoon portion of the July 1982 Licensing Examination, two of the four possible responses, "A" and "C" were credited. Candidates like Petitioner Cauthen who answered "B" received no credit for the question. Petitioners Harden and Cauthen have requested Chapter 120.57 hearings with respect to their January and July 1982 licensing examinations and the review, grading and agency action with respect to same. Although request for production was specifically made for said packages of information with respect to the January 1982 and July 1982 licensing examinations in February, 1983, no packages have been produced to date. DPR destroyed the Petitioners' examination booklets before the end of the two year period immediately following each of the 1982 examinations. That Section 455.217 of the Florida Statutes requires the Board "by rule" to designate areas of competency to be covered by each licensing examination. That Section 455.217 states that the Board shall "by rule specify the general areas of competency to be covered by each examination, the relative weight to be assigned in grading each area tested, and the score necessary to achieve a passing grade." That the amendment of Rule 21GG-6.01 (2) required candidates to commit to memory portions of the electrical code, accounting, law, worker's compensation rules, federal employer's tax guide, A1A General Conditions, business practices; legal and insurance requirements. In the July 1982 Electrical Contractors' Licensing Examination candidates were given one hour more time in which to take the examination than was given during the January 1982 examination. Rule 21GG-6.01 (4) was amended in conjunction with Rules 21GG-6.01 (1)(a), (2) and (3) at a board hearing in March, 1981. Rule 21GG-6.01 (4) raised the passing score on the January 1982 Electrical Contractors' Examination and the July 1982 Electrical Contractors' Examination from 70 to 75. The amendment to Rule 21GG-6.01 (4) required that electrical contractors pass the certification examination with a score of 75 percent, whereas all other construction industry licensing board contractors licensed by the Department of Professional Regulation (13 other categories) were merely required to achieve a 70 percent pass score. That Section 455.217 of the Florida Statutes provides that "The board shall make rules providing for reexamination of any applicants who have failed the examination." That DPR did not and has never provided Petitioners, their attorneys or agents with their own examination booklets for the January and July 1982 Electrical Contractors' Licensing Examinations. DPR has provided Petitioners only with copies of "master" examinations for their review. That Section 455.217 of the Florida Statutes provides that the Board shall make available an examination review procedure for applicants. That the Respondents produced Notice of Destruction of Examination Booklets and Other Examination Materials dated April 14, 1982, which allegedly evidences destruction of Petitioners' examination booklets for the January 7, 1982, Examination on April 14, 1982. That the Respondents produced Notice of Destruction of Examination Booklets and Other Examination Materials dated October 15, 1982, which allegedly evidences destruction of Petitioners' examination booklets for the July 13, 1982, Examination on October 15, 1982. The findings in paragraphs 1 through 33, immediately above, are based directly on the stipulations of the parties, most of which stipulations were memorialized at the beginning of the hearing. The findings in the following paragraphs are based primarily on testimony and exhibits, but some of them are also based in whole or in part on stipulations. In the findings which follow there are certain to be at least some repetitious findings in the course of putting matters into context and making additional findings which are related to some of the stipulated findings. I have tried to avoid all unnecessary repetition, but a certain amount is necessary for clarity and a certain amount is unavoidable due to the sheer size of the task at hand. Findings on background matters and on matters relating to more than one rule Both of the Petitioners in this rule challenge proceeding are individuals who have applied to the Electrical Contractors' Licensing Board for licensure as certified electrical contractors. Both of them have been approved to sit for the licensure examination. Petitioner Harden took the Board's licensure examination on each of the following occasions: July 1981, January 1982, July 1982, and January 1983. The Board has not given him a passing grade on any of those' four examinations. Petitioner Cauthen took the Board's licensure examination on two occasions: January 1982 and July 1982. The Board has not given her a passing grade on either of those two examinations. The grade notifications received by these Petitioners show, inter alia, that Petitioner Cauthen received a grade of 73 on the July 1982 examination. Both of these Petitioners have presently pending formal proceedings under Section 120.57(1), Florida Statutes, in which they are challenging various matters related to the preparation of, administration of, scoring of, and inherent validity of the Board's January 1982 and July 1982 licensure examinations. The Electrical Contractors' Licensing Board certification examination is different from a master electrician examination. However, the two examinations are in many ways similar because of the overlap in the nature of the subject matter to be tested on both examinations. The passing score or cut score on the local master electrician examination administered by Dade County is 70 percent. Most of the questions on that examination are about the National Electric Code. Part of the Dade County master electrician exam is closed book. Prior to 1972, persons wishing to engage in electrical contracting in the state of Florida were required to be licensed by the local governments in the areas in which they sought to operate. Since 1972, persons wishing to engage in electrical contracting in the state of Florida must be licensed by a unit of local government or by the Electrical Contractors' Licensing Board. Persons who are licensed by the Electrical Contractors' Licensing Board are known as certified electrical contractors. A certified electrical contractor can engage in electrical contracting anywhere in the state of Florida without local licensure. Persons who are licensed by one or more local governments (typically a municipality or a county) are known as registered electrical contractors. Registered electrical con-tractors are licensed to engage in electrical contracting only in the geographic areas encompassed by the boundaries of the local government entities that issued their local licenses. There is, however, a certain amount of reciprocity from one local government to another. Certification as a state certified electrical contractor does not authorize the electrical contractor to work as an electrician, although a certified electrical contractor can pull permits. A person does not have to be a licensed electrician in order to become a state certified electrical contractor, although many electrical contractors are also licensed as journeyman or master electricians. Candidates for the electrical contractor licensure examination have a great $ variety in the nature and scope of their background and experience. This variety in background and experience is among the reasons which cause testing for minimum competence as an electrical contractor not to be an exact science. As between certification and registration, certification by the Electrical Contractors' Licensing Board is not a major factor in the ability to compete for business. A person who has a statewide certification can save some money by avoiding the expense of obtaining local competency cards, but as a general rule a registered electrical contractor can work just about anywhere in the state due to reciprocity. Especially, a person who has passed a Block master electrician examination or a Block local electrical contractor's examination can work just about anywhere in the state because most counties in Florida accept the Block examination. Registered electrical contractors regularly compete for business with certified electrical contractors. For example, Petitioner Cauthen's company is presently licensed in Monroe, Dade, and Broward counties, where it does a considerable amount of business. Petitioner Cauthen's company bids on lots of contracts and submits bids in competition with both registered and certified electrical contractors. Thus, the company is already in competition with certified electrical contractors. Similarly Board Member Isaacs, whose certified company does 90 percent of its work in Duval County, has regular competition from registered electrical contractors. Board Member Isaacs competes with approximately 250 electrical contractors. Of that 250, approximately 175 are registered and the others are certified. A much bigger factor in competition than the registration versus certification issue is the cost of doing business in more than one area or the cost of doing business at a location that is distant from one's primary base of operations. Only about fifty electrical contracting firms regularly compete for business over the entire state of Florida. Accordingly, a restriction on the number of persons licensed by the Board as certified electrical contractors would have minimal, if any, limitation on the competition faced by those members of the Board who are certified electrical contractors. The Electrical Contractors' Licensing Board admin-istered its own certification licensure examination from 1972 through 1980. During that period of time the Board administered the examination a total of twenty-two times. During those twenty-two sessions of Board administered examinations, a total of 824 candidates sat for the examination, of which 392 were successful. Although the percentage of candidates who were successful on a particular Board administered examination ranged from a low of 20.7 percent to a high of 78.8 percent, the average passing rate of all candidates on all twenty- two of the Board administered examinations was 47.57 percent. (By way of comparison, it is interesting to note that the percentage of candidates who are successful on the Block master electrician examination is between 30 percent and 35 percent of those who take the examination.) Further, on fifteen of the twenty-two occasions on which the Board administered the examination, the percentage of candidates who were successful was 50 percent or less. In 1979, the regulation of professions and occupations was reorganized. The Department of Professional Regulation (the Department) was created as an umbrella agency over numerous boards, including the Electrical Contractors' Licensing Board and the Construction Industry Licensing Board. The responsibility for examining applicants, which in the case of the Electrical Contractors' Licensing Board had been handled by the Board members themselves, was taken over by the Office of Examination Services (O.E.S.). Toward the end of 1980 and the beginning of 1981, O.E.S. did not like what the Board was doing and the Board did not like what O.E.S. was doing. There was a general lack of trust between the two entities, there was poor communication between the two entities, and the relationship between them at that time might best be described as estranged. Although both entities made efforts at cooperation with the other, such efforts were not always effective or well received. As a result of the poor communication between the two entities, certain personal concerns over Board actions by individuals within the O.E.S. were never communicated to the Board (and therefore could not be answered. 4/ The first examination administered by the O.E.S. was in January 1981. Of the fifty candidates who took the January 1981 examination, forty-six were successful. This was a passing rate of 92 percent. In view of the Board's experience when it was administering its own examination, the Board was understandably surprised and concerned when the percentage of candidates passing the first O.E.S. administered examination was almost twice the average passing percentage on the Board administered examinations. The Board had no evidence that the overall qualifications or capabilities of the applicants examined by O.E.S. at the January 1981 examination were any higher than the capabilities of those examined previously. To the contrary, a review of their qualifications based on the information in their applications indicated that their qualifications were substantially the same as those of previous candidates tested by the Board. Also, the admission standards for the January 1981 examination were the same as those for prior examinations. The Board concerns about the O.E.S. administered January 1981 examination included the following matters, among others, memorialized at a Board meeting discussing the examinations: The candidates had been allowed to take any reference material desired into the exami- nation. Previously the Board had allowed only reference books listed as part of the application form. There was a very high pass rate in compar- ison to all past examinations. There appeared to be too few calculation questions. The questions had been placed on the paper in sequence with the reference book materials. There was a possibility that notes were taken into the examination which would have given advantage to the candidates. The Board's Examination Committee had been under the assumption that the entire examination would be presented for review of the questions. Instead the only questions presented were ones questionable under Department criteria for measuring competency. It was explained, from past experience, that persons conducting the seminars for exams had obtained the entire exam content from persons taking the examination in the past. It had been done with a camera which was taken into the examination. The Board, when administer- ing the examination, had been very strict on what was used in performing the examination. In the last exam the security was very poor. The Board felt there were so few calculation questions on the O.E.S. administered examination in January of 1981 that the examination was not a proper or sufficient examination. This was due at least in part to the fact that O.E.S. had difficulty covering all areas of the examination with the desired number of questions because the O.E.S. bank of questions was very limited at that time. Previous Board examinations were open book with certain reference material allowed. When the Office of Examination Services took over the administration of the January 1981 exams, it sent to the candidates a form letter adapted from the Construction Industry Licensing Board. This form letter instructed the candidates that they would be able to bring into the examination any notes and other materials desired. This was contrary to the past practice of the Board and contrary to the Board's wishes. The Board was very frustrated and concerned about the circumstances which led up to the January 1981 examination because the O.E.S. had refused to allow the Board to have any input into the examination and the Board thought the O.E.S. had come up with an apprentice level examination. In the Board's opinion, the January 1981 examination prepared by O.E.S. clearly tested at a level less than that of minimum competency. And, although O.E.S. did not agree with the Board that the January examination was invalid, O.E.S. did agree that the examination was a lot easier than they had thought it would be. The Electrical Contractors' Licensing Board did not conduct any formal studies or formal analysis prior to making the 1981 amendments to Rule 21GG-6.01 changing the format of the examination and the cut score. However, the Board's Rules Committee did a great deal of work on the matter and the matter was discussed extensively at Board meetings. In this regard it is important to keep in mind that the Board members (with the exception of the two lay members) were all experienced practicing electrical contractors who were familiar with the requirements of day-to-day electrical contracting. The Board's decision to change part of the examination to closed book was due to a concern that people with no electrical knowledge could become licensed if the examination was all open book and had a low percentage of technical questions. The Board felt that more than half of the examination should be technical questions. The closed book National Electrical Code questions were for the purpose of testing what a person with experience in the field of electrical contracting should know about everyday matters. The primary, if not the sole, motivation for the Board's 1981 amendments to Rule 21GG-6.01 was the Board's concern about the quality of the O.E.S. administered examination and the Board's feeling that, due to its estranged relationship with O.E.S., the only vehicle through which it could effectively influence the quality of future examinations was through rule- making. Those amendments were not motivated by any desire on the part of the Board to restrict competition in the field of electrical contracting. During the past few years the job requirements of an electrical contractor have remained substantially the same. Block and Associates (hereinafter "Block") is a company based in Gainesville, Florida, that writes licensure examinations for contractors in various trades. Block writes such examinations for cities, counties, and states. Block has prepared electrical contracting licensure examinations for the states of Florida, Georgia, and Oklahoma, as well as for St. Johns County, Florida. Block prepares licensure examinations for just about anything that involves electricity, including such things as electrical contracting, electrical journeyman, electrical-master, low voltage, elevator, burglar alarm and fire alarm Block prepares master electrician examinations for over one hundred governmental entities. The Block electrical master exams and the Block electrical contractor exams use some of the same questions. Subsequent to the 1982 examinations which form the gravamen of these Petitioners' related proceedings under Section 120.57, Florida Statutes, the ECLB has contracted with Block and Associates to prepare the Board's certification licensure examination. The Board has never said anything to Block about wanting to achieve any particular passing rate or wanting to have any particular percentage of the candidates taking the examination achieve a passing score. The Board has never interfered with Block's autonomy in the preparation of the electrical contractor exam. All of Block's electrical contractor examinations include business questions. It is important to test for business skills because an electrical contractor who gets into business trouble may start cutting corners and cutting corners is a hazard to the public. Some of Block's electrical contractor examinations also have specific sections on safety. Block always does a post-administration statistical analysis of the examinations questions it uses. The reason for this is to find out if anything is wrong with the questions; in other words, to determine whether the questions are valid. The computer program used by Block to check the validity of its tests was prepared for Block by a professor in the College of Education at the University of Florida. Block uses a computer to conduct a validity analysis of each test it administers. Findings regarding Rule 21GG- 6.01(1)(c) of the Electrical Contractors' Licensing Board Rule 21GG-6.01(1)(c) of the Electrical Contractors' Licensing Board reads as follows (with the portion challenged by the Petitioners underscored): The areas of competency to be covered by the certification examination shall be as follows: (a) *** (c) Safety, which shall include but not be limited to, OSHA regulations, life safety codes, and the Fire Safety Code (NFPA). Prior to the December 1981 amendment which is challenged here, Rule 2100-6.01(1)(c) read as follows: The areas of competency to be covered by the certification examination shall be as follows: (a) *** (c) Safety, which shall include but not be limited to, OSHA regulations, life safety codes, and the fire safety code (NFPA 72A, B, C and D). The Board's purpose in amending the rule to delete the reference to "72A, B, C and D" was to make it clear to candidates for examination that the Board' intended to include in the safety portion of its examinations questions from portions of Volume Seven of the NFPA other than Chapters 72 A, B, C, and D of that volume. Volume Seven of the NFPA contains quite a bit of material in addition to the material included in Chapters 72 A, B, C, and D of that volume. In view of the modifying phrase "shall include but not be limited to," the amendment to delete the reference to Chapters 72 A, B, C and D was not a necessary prerequisite to the use of examination questions based on other portions of Volume Seven of the NFPA. The Board could ask the same examination questions before and after the amendment. Several weeks before each examination, the Department of Professional Regulation sent all candidates for examination, including these Petitioners, a notice to appear for the examination. The notice to appear included a list of reference books on which the examination was to be based. That list of reference books contained a specific reference to Volume Seven of the NFPA. All sixteen volumes which comprise the entire NFPA were not listed as reference books. Neither of these Petitioners demonstrated that he or she was genuinely confused or misdirected by the amendment to Rule 21GG-6.01(1)(c). Neither Petitioner contended he or she had studied all sixteen volumes of the NFPA. Instead, both Petitioners studied from the books on the reference list. Neither Petitioner has an application for the examination pending; neither claims an intention to take the examination again. Findings regarding Rule 21GG- 6.01(1), (2), (3), and (4) of the Electrical Contractors' Licensing Board At the time of the examinations which are the subject of these Petitioners' related cases under Section 120.57, Florida Statutes, Rule 21GG- 6.01(1), (2), (3), and (4) of the Electrical Contractors' Licensing Board read as follows, in pertinent part: The areas of competency to be covered by the certification examination shall be as follows: Technical, which shall include, but not be limited to, electrical calculations, estimating, designs, and electrical schematics; and which shall be divided into two (2) separate areas, one containing electrical calculations, the other containing Code-related questions not requiring calculations. General business, which shall include but not be limited to, problems relating to accounting, law, insurance, workers' compen- sation and social security; The format of the examination shall be as follows: The portions of the examination con- taining the electrical calculations and $ safety questions shall be open book. The applicant is responsible for bringing and may.. use during these portions the applicable code books, reference materials as approved by the Board, and noiseless mechanical or non- mechanical instruments he wishes to use. The portions of the examination con- taining the noncalculation technical questions and all business questions shall be closed-book. The applicant will not be permitted to use the reference materials listed in 2(a) or any other reference materials. Security measures as set forth by the Department shall be followed during both portions of the examination. The relative grading weight to be assigned to each area of competency shall be approximately as follows: Technical - 60 percent Electrical Calculations-40 percent of Test Code-related questions-20 percent of Test General business - 30 percent Safety - 10 percent An applicant shall be required to achieve a score of a general average of not less than seventy-five percent (75 percent) in order to pass the examination and be certified for licensure. There shall not be a practical or clinical examination. The language quoted above is the language of the relevant portions of the rule as it read after the March 1981 amendments to the rule. The principal amendments in March of 1981 may be summarized as follows: (1) changes which divided technical questions on the examination into two separate areas, (2) changes which made part of the examination open book and part closed book [it was previously all open book, and (3) a change in the minimum passing score from 70 percent to 75 percent. These Petitioners challenge the validity of all three of the principal amendments. As originally proposed, the March 1981 amendments to subsection (2)(a) of Rule 21GG-6.01 would have permitted a candidate to bring "any notes" to the open book portion of the examination. During the adoption process the words "and any notes" were deleted from the amendment by the Board. The words "and any notes" did not appear in Rule 21GG-6.01 prior to the 1981 amendments. Nor was it the practice of the Board prior to 1981 to permit candidates for examination to take any notes into the examination when it was administered by the Board. Findings regarding examination content and relative grading weights In November of 1983 the Electrical Contractors' Licensing Board again amended Rule 21GG-6.01(3) to change the relative grading weights of the areas of competency to be tested. After November of 1983 the relative grading weights of the areas of competency were as follows: Technical increased from 60 percent to 65 percent; electrical calculations increased from 40 percent to 45 percent; code related questions remained 20 percent of the examination; general business was decreased from 30 percent to 25 percent of the examination; and safety remained 10 percent of the examination. 5/ The licensing examination given by the Board has historically contained materials testing knowledge of electrical work and also materials testing the business qualifications of the applicants. Since the inception of the Board's licensing exam, the electrical work portions have contained electrical calculations. Throughout the time the Board has been administering licensure examinations for certification, the percentages of the examination devoted to particular subject areas have been based upon the Board's belief as to the appropriate levels of knowledge needed by a certified electrical contractor. The weighting of the examination was equitable when it was weighted 60 percent technical, 30 percent business, and 10 percent safety, and it is still equitable now that it is weighted 65 percent technical, 25 percent business, and 10 percent safety. The Board changed to the current percentages at the suggestion of Block and Associates. The Board's examination has had similar ratios for a long time, at least as far back as 1971. The proportions of the examination devoted to each particular subject matter did not change due to the March 1981 amendments. With the exception of the January 1981 examination put together by O.E.S., electrical calculations had historically been 40 percent of the test. Since the Board felt this percentage should be maintained, and since it was necessary to divide those calculations from the other portion of the technical section (since one would be tested in the closed book portion and the other in the open book portion), the division of the technical part of the exam was set out in the rule. On an electrical contractor licensure examination it is appropriate to include in the area of "technical" questions, questions dealing with the following subjects: electrical calculations, parts of estimating, and designs and electrical schematics. In the "business" portion of an electrical contractor examination, it is appropriate to include questions dealing with the following: accounting, worker compensation, and social security. Findings regarding closed book versus open book examinations Local licensing examinations had, in 1981, included a closed book portion. Local licensing examinations and state licensing examinations given in other states still include such a closed book portion. The basic Block examination has always included a closed book portion. Closed book examinations are not all that unusual in occupational and professional licensure testing. The CPA an Nursing examinations are closed book. Also, virtually all of the electrical examinations administered by Block and Associates include a closed book portion. The advantage of a closed book examination technique is that it is easier to identify the candidates who have had practical experience in the subject matter being tested. A closed book examination is a better test for field experience and for general knowledge of the subject matter being tested. Certain questions known as "anchor" questions was given both in the open book examination before the format change and in the closed book section after the format changed. The relative performance by candidates on the same questions in the two formats shows the format change not to have prejudiced the candidates. Further, an examination analysis of the results of the closed book portions of the ECLB examinations reveals that, based on candidate performance, the closed book portion of the examinations was easier than the open book portion. This also shows that the format change did not prejudice the candidates. Findings regarding "cut scores" From 1972 until the rule changes in 1981, the minimum passing score, or "cut score," set by the Board for the certification licensure examination was 70 percent. Cut scores of both 70 percent and 75 percent are very common cut scores for licensure examinations in the fields of electrical contracting, journeyman electrician, and master electrician, as well as licensure examinations in other occupations and professions. Many local licensure examinations have a 75 percent cut score. Standing alone, the cut score to be applied to a future examination is somewhat lacking in specific significance. A cut score must be considered in relation to the content of the examination to which it is to be applied. All of the testing experts who testified at the hearing agreed that setting an examination cub score is an inherently arbitrary decision. In the final analysis it is a judgment call that cannot be made with scientific exactitude and is unavoidably arbitrary to a certain extent. As explained by one of the Petitioners' testing experts, even though statistical methodology is available for the determination of cut scores, to a large extent the cut score is a random decision, the validity of which can be assessed only in relationship to the specific testing instrument to which it is applied. For example, during the course of Block's preparation of its first licensure examination in 1958 for the City of Ormond Beach, when it came time to decide what cut score to use, the five experts who were preparing the examination talked it over and decided that 75 percent was the equivalent' of a good. "C" when compared to academic grades, and they decided to use 75 percent as their cut score. It is not possible to be much more scientific or precise than that in setting cut scores, especially when one has a group of candidates of varied experience and background taking the examination. A skilled test preparer can construct an examination of equal validity within a range of cut scores from 65 percent to 80 percent if the test preparer knows at the time of constructing the examination what the cut score will be. With any cut score in that range, a skilled test preparer can raise or lower the percentage of candidates who will actually pass the examination by careful selection of the questions to be used on the examination. By using easy questions, the person preparing the examination can insure that more people will pass a test with a high cut score. Similarly, by using hard questions, the person preparing the examination can insure that fewer people will pass an examination with a low cut score. On some standardized licensure examinations the cut score is determined after the examination results are examined. In other words, the examination results are "curved" based on the performance of the group of candidates who take a particular session of the examination and the actual cut score is determined in relationship to the raw scores achieved by each particular group of candidates. This flexible method of setting cut scores seems to work well on standardized examinations which are administered to large numbers of candidates, but it is questionable whether it would work very well with small groups of candidates. The Board's examinations have historically been administered to small groups of candidates. Findings regarding the effects of changes on testing instruments. In order for a licensure examination to accomplish the purpose for which it is administered, it should, of course, be valid and reliable. Several of the Petitioners' expert witnesses described what might be characterized as the best way to make changes to an examination or as the ideal way to make changes to an examination. Nevertheless, the manner in which the Electrical Contractors' Licensing Board went about making the changes to the examination which were incorporated in the March 1981 rule changes was a reasonable and logical response to the circumstances faced by the Board. It was perhaps not the best way to have resolved matters, but it was a reasonable response. The Board's action in this regard was not arbitrary or capricious. Even the Petitioners' most persuasive expert in the field of testing conceded that if the content experts (in this case the Board members) felt that given a specific content domain and given a specific item bank the testing instrument measured less than minimum competency, a logical response would be to raise the cut score. Another logical reaction to that situation would be to change a portion of the examination from open book to closed book. Both actions would be reasonable and logical responses to that type of problem. Where there has been a modification in the format of a testing instrument, an accepted methodology for analysis of the effects of format modification is the use of repetitive questions from prior testing instruments. These repetitive questions are known as "anchor" questions. If a statistically significant deviation in performance on the anchor questions is demonstrated, the format modification requires further analysis through other statistical tools. However, if an analysis of the anchor questions shows inconclusive or mixed results on the anchor questions (such as equal or improved candidate performance on those questions), it may then be concluded that the format change is not significant to candidate performance and may be an appropriate format change relative to the profession or occupation being examined. Content validity is the most practical' measure of the validity of a licensure examination. The content validity of an examination depends on the extent to which the questions on an examination may be accepted as representative of performance within a specifically defined content domain of which the examination instrument is a sample. Content validation would be the best way to determine the validity of the examinations which were administered in 1982 pursuant to the March 1981 rule amendments. One method of establishing content validity is to have a group of experts identify the domains of knowledge that are applicable to the profession or occupation being tested. The establishment of content validity relies heavily on the opinions and judgments of people who are experts in the relevant profession or occupation. Expert judgment plays a integral part in developing the definition of the content domain to be tested. To the extent that the content domain of Boar examinations was established by the March 1981 amendments to Rule 21GG-6.01(3), the content domain was established on the basis of the expert judgment of persons who were experts in the field of electrical contracting, namely, the members of the Electrical Contractors' Licensing Board. The relative grading weights established in the 1981 amendments to Rule 21GG-6.01(3) have at least a possible correlation to electrical contractor competence. In fact, the persuasive expert testimony establishes that those relative grading weights did bear a reasonable relationship to electrical contractor competence. The current slightly different relative grading weights also bear a reasonable relationship to electrical contractor competence even though there has been no recent change in the nature of the responsibilities of electrical contractors. In this regard it must be remembered that testing is not an exact science and no test for minimum competency can ever be expected to be an exact model of actual work experience. The best that can be expected is a reasonable model. There are various accepted methodologies for the creation of standardized examinations, adherence to which tends to enhance the likelihood that the examination instrument finally produced will be valid and reliable. However, adherence to those accepted procedures is not a necessary prerequisite to creation of a valid and reliable examination. This is particularly true when one is going to be testing small groups of people with varied backgrounds instead of the massive groups of people with similar backgrounds for whom standardized tests are more typically designed. In the final analysis the only practical and reliable measure of the validity of an examination is by statistical analysis of the examination after it has been administered. Given the nature of the circumstances faced by the Board at the time of the March 1981 amendments to the examination format and cut score, given the nature of the pool of candidates to be examined, given the nature of the changes contemplated by the Board, and given the very nature of the process of testing for minimum competency--which involves perhaps as much art as it does science-- there is no study or data which would have been particularly useful to the Board in helping to determine exactly what the effect of their changes would be. Such effects can only be determined or measured with any degree of accuracy after the administration of an examination that incorporated the changes. Following the administration of such an examination, it is possible to perform a statistical analysis of all questions used on the examination and to eliminate or give credit for any questions which are shown by statistical analysis to be invalid or unreliable. This is precisely the process that is used by Block in the validation of their examinations and is an accepted testing procedure. It would have served no useful purpose to have conducted a trial run of an examination using licensed certified electrical contractors as a test group for the new examination format. First, it would be virtually impossible to try to put together an accurate cross-section of certified electrical contractors to use as a test group. Second, one would expect them all to pass the examination, so when they did so nothing of value would have been learned. Finally, the administration of such a trial run would risk the possibility of compromising examination question security. One aspect of accepted methodology for the preparation of standardized examinations is the definition of the content domain of the examination, i.e., a determination of what knowledge is essential to demonstrate that the candidates for licensure are minimally competent. This aspect of examination preparation is often accomplished by performing a formal job analysis, which is, in essence, a study of all of the usual tasks performed by a person engaged in the occupation or profession to be tested, including an evaluation of the relative importance of each of those tasks to minimum competence. The content domain can also be defined on the basis of the judgment of a group of experts in the occupation or profession to be tested. With the exception of the lay members who were added in recent years, all of the members of the Electrical Contractors' Licensing Board are, and have been, persons certified to engage in electrical contracting in the state of Florida and actively engaged in the electrical contracting business. Therefore, at all relevant times all of the professional members of the Board had extensive personal knowledge of what was involved in the practice of electrical contracting, which personal knowledge was as useful in defining content domain as would have been a formal job analysis. (In this regard it is important to note that even with the addition of lay members to the Board in recent years, the experienced professional members have continued to constitute a substantial majority of the Board.) Findings regarding Rule 21GG-6.0 of the Electrical Contractors' Licensing Board Rule 21GG-6.03 of the Electrical Contractors' Licensing Board reads as follows: An examinee is entitled to review his examination questions, answers, papers, grades and grading key used in the certification examination; however, no applicant may copy any materials provided for his review. Such review shall be conducted during regular business hours, in the presence of a representative of the Board at the Board's official headquarters. If, following the review of his exami- nation, an examinee believes that an error was made in the grading of his examination, or in the evaluation of his answers, he may request the Board to review his examination. Requests for Board review must be in writing, state with specificity the reason why review is requested, and be received within thirty (30) days after the examinee received notice that he failed the examination. (3) Upon receipt of a request for Board review, the examination shall be reviewed by the Board at the next regularly scheduled Board meeting. If it is found that an error was made, the grade received by the examinee may be adjusted to reflect the correction. The examinee shall be notified of the final decision. The Petitioners' only challenge to Rule 21GG-6.03 is predicated upon their concern that it may be misapplied to them by the Board. Findings regarding Rule 21-11.11(3) of the Department of Professional Regulation as it existed prior to the October 1982 amendments Prior to the October 1982 amendments, Rule 21-11.11(3) of the Department of Professional Regulation read as follows: (3) Examination grade reviews shall be conducted at a site designated by the Department. The candidate and his/her attorney or other qualified representative shall be allowed to attend the examination grade review. A candidate may attend only one review per examination administration. The above quoted version of the rule was in effect at the time of the Petitioners' reviews of their January and July 1982 examinations. Findings regarding Rule 21-11.11(3) of the Department of Professional Regulation, as amended in October of 198 Rule 21-11.11(3) of the Department of Professional Regulation, as amended in October of 1982, reads as follows, in pertinent part: A candidate taking a Department exam- nation may request and receive an appointment for review until such times as the exam- ination records are destroyed in accordance with Chapters 455, 119 and 267, Florida Statutes. However, unless otherwise provided by a rule of the appropriate Board within the Department, candidates may review their exam- ination for the purpose of filing objections to the examination for the Board's consideration under the following conditions and time frame: Within thirty (30) days of the date of the grade notification letter the candidate shall notify the Department of his/her desir to review the examination for the purpose of filing objections for consideration. Such review shall have been completed within the next thirty (30) days after the first thirty (30) days defined in (a) above. At the examination review, the candi- date shall be permitted to record on forms provided by the Department any and all objections to the examination the candidate desires the appropriate Board to review. Such forms shall remain in the custody of the Office of Examination Services for presen- tation to the appropriate Board at the next available Board meeting as an official agenda item. The candidate shall not copy" questions from the test booklet. The candidate may write on a separate paper in the presence of the Office of Examination Services employee, any objection or question he/she has to the written examination. The candidate shall leave the written objections and questions with the Office of Examination Services employee when he/she leaves the review room but she/she shall be permitted to leave with a form listing the question numbers he/she finds to be controversial. In the instance of a written exami- nation, all objections will be presented to the appropriate Board for consideration. Upon completion of the Board's review of written examination items the Department shall notify the reviewing candidate of the Board's decision. If the Board does not concur with the candidate's objections then the candidate will be notified in writing of this and the thirty (30) day appeal time per- iod shall begin to run from the date of this notice. Findings regarding Rule 21- 11.11(13) of the Department of Professional Regulation (later renumbered as Rule 21-11.11(3)(1)j Rule 21-11.11(13) of the Department of Professional Regulation, which was later renumbered as Rule 21-11.11(3)(1), reads as follows: If the consultant finds that the original grade was not rendered in accordance with the grading criteria, then he/she twill regrade that portion, or the entire examination, whichever is appropriate, pursuant to appli- cable statutes and rules. If it is not possible to regrade the examination, the candidate will be allowed to retake the examination at no charge. It is clear from the text of Rule 21-11.11(3)(j) that the language of Rule 21-11.11(3)(1) quoted immediately above is intended to apply only to practical examinations. It does not purport to apply to written examinations. These Petitioners took only written examinations, they have not taken any practical examinations. Findings regarding Rule 21-11.14 of the Department of Professional Regulation Rule 21-11.14 of the Department of Professional Regulation reads as follows, in pertinent part: Unless otherwise provided by a rule of the appropriate board within the Department, Security and Monitoring procedures shall be as follows: (10) Disposition of Test Booklets. Secured processing of Answer Sheets Test booklets for the state devel- oped examinations shall be disposed of and filed in accordance with the following procedure: 1. Ten copies of the examination book- lets shall be retained for ninety (90) days. This procedure shall insure that there are adequate copies of the booklets available for the board to review in their review of preliminary analysis of the examination and review sessions if review responsibilities are retained by the board. After the above defined ninety (90) day period the retention schedule shall change from ten copies to four copies. In the event any irregularity occurs during the examination with any state developed booklet, it shall be the examination supervisor's responsibility to prepare a detailed report of such irregularity and to retain the booklet in question in the secured files for a period of ninety (90) days. Destruction of examination booklets and related materials described in B.2.b., above: When in accordance with procedure, it is appropriate to destroy test booklets, it shall be the Archives's responsibility to schedule such destruction. In all instances there shall be evidenced in writing by the examination supervisor and a witness on a form which shall evidence the date of destruction and the official in charge of such destruction and a witness. When examination booklets are being prepared by the Department of Professional Regulation prior to an examination, the booklets are carefully inspected to make sure that all of the booklets are identical. Following that inspection the booklets are sealed and stored in a secure place in order to insure, among other things, that no changes are made to any of the examination booklets before they are handed out to the candidates. After an examination is given, the Department of Professional Regulation retrieves all of the examination booklets, including all booklets that were used by all of the candidates, and retains them in a secure place until the excess booklets can be destroyed. The examination supervisor selects the booklets which are to be retained from among the booklets that were not handed out to the candidates at the examination. In the normal course of events all of the examination booklets that were actually handled by the candidates at the examination are destroyed within a very few months of the date of the examination. With the exception of the examination booklet of one other candidate (which exception is not relevant to the disposition of this rule challenge proceeding), all of the examination booklets which were handed out to candidates during the examinations taken by these two Petitioners were destroyed approximately 90 days after each of the examinations. When such destruction took place, the Department retained copies of the examination booklets which had not been used by any candidate, which copies were identical to the copies that had been handed out to the candidates during the examination. As part of the examination instructions, all candidates for examination are advised not toe write anything in their examination booklets because all of the booklets used by the candidates will be shredded. They are specifically told to do all of their computations on sheets of work paper that are provided to them at the examination. All candidates are specifically told that the only things they turn in that will be saved are their answer sheets and their sheets of work paper. Findings regarding "unwritten" and "unpublished" rules of the Electrical Contractors' Licensing Board. The findings of fact immediately below relate to the eight "unwritten" and "unpublished" alleged rules of the Electrical Contractors' Licensing Board which are described in subparagraphs (F), (O), (P), (V), (X), (Y), (Z), and (FF) at pages 1 through 5 of the Petition. (The descriptions of these alleged rules are quoted at pages 2 through 6 of this Final Order.) One of these "unpublished" rules is alleged to be a rule to the effect that the Board requires more than minimum competency in order for a candidate to receive state certification and licensure as an electrical contractor. The other seven "unwritten" or "unpublished" rules of the Electrical Contractors' Licensing Board alleged in the Petition are all related in one way or another to the Board's alleged free form actions in the course of its review of the two examinations which have been challenged by these Petitioners in related proceedings under Section 120.57(1), Florida Statutes. None of the evidence regarding these matters proved the existence of an agency statement of general applicability purporting in and of itself to have the direct and consistent effect of law. For reasons which are explained in the Conclusions of Law portion of this Final Order, the evidence is insufficient to establish that the Electrical Contractors' Licensing Board has any "unwritten" or unpublished5? rules such as those described in subparagraphs (F), (O), (P), (V), (X), (Y), (Z), and (FF) of the Petition. 6/ Findings regarding "unwritten" and "unpublished" rules of the Department of Professional Regulation The findings of fact immediately below relate to the six "unwritten" and "unpublished" alleged rules of the Department of Professional Regulation which are described in subparagraphs (G), (J), (1), (L), (Q), and (R) at pages 1 through 5 of the Petition. The descriptions of these alleged rules are quoted at pages 2 through 6 of this Final Order.) These six "unwritten" or "unpublished" alleged rules of the Department of Professional Regulation are all related in one way or another to actions of functionaries of the Department in the course of performing the Department's role in the administration and review of the two examinations which have been challenged by these Petitioners in related proceedings under Section 120.57(1), Florida Statutes. None of the evidence regarding these matters proved the existence of an agency statement of general applicability purporting in and of itself to have the direct and consistent effect of law. For reasons which are explained in the Conclusions of Law portion of this Final Order, the evidence is insufficient to establish that the Department of Professional Regulation has any "unwritten" or "unpublished" rules such as those described in subparagraphs (G),(J), (I), (L), (Q), and (R).7 Findings regarding "unwritten" and "unpublished" rules of both the E1Electrical -Contractors' Licensing Board and the Department of Professional Regulation The findings of fact immediately below relate to the ten "unwritten" and "unpublished" alleged rules of both the Electrical Contractors' Licensing Board and the Department of Professional Regulation which are described in subparagraphs (M), (N), (S), (T), (U), (AA), (BB), (CC), (DD), and (EE) at pages 1 through 5 of the Petition. (The descriptions of these alleged rules are quoted at pages 2 through 6 of this Final Order.) These ten "unwritten" or "unpublished" alleged rules of both the Electrical Contractors' Licensing Board and the Department of Professional Regulation are all related in one way or another to actions allegedly taken by the Board and the Department in the performance of their respective functions related to the administration and review of the two examinations which have been challenged by these Petitioners in related proceedings under Section 120.57(1), Florida Statutes. None of the evidence regarding these matters proved the existence of an agency statement of general applicability purporting in and of itself to have the direct and consistent effect of law. For reasons which are explained in the Conclusions of Law portion of this Final Order, the evidence is insufficient to establish that both the Electrical Contractors' Licensing Board and the Department of Professional Regulation have any "unwritten" or "unpublished" rules such as those described in subparagraphs (M), (N), (S), (T), (U), (AA), (BB), (CC), (DD), and (EE) of the Petition.8
The Issue Whether Petitioner's challenge to the failing grade he received on the contract administration portion of the October 2000 General Contractor Examination should be sustained.
Findings Of Fact Based upon the evidence adduced at hearing and the record as a whole, the following findings of fact are made: Petitioner sat for the contract administration portion of the Florida certification examination for general contractors administered in October 2000 (Contract Administration Examination). The Contract Administration Examination consisted of 60 multiple-choice questions of equal value, worth a total of 100 points. To attain a passing score on the Contract Administration Examination, candidates needed to receive a total of 70 points. Of the 378 candidates who took the Contract Administration Examination, 156 received passing scores. Petitioner was not among this group of successful candidates. He received a failing score of 66.67 on the examination. Question 2 of the Contract Administration Examination was a clear and unambiguous multiple-choice question that required the candidate to determine, based upon the information given, on what workday (not calendar day) the pouring of concrete footings for a residential construction project would begin. There was only one correct answer to this question. Approximately 50 percent of the candidates chose this correct response. Petitioner chose another answer that was clearly incorrect because it represented the calendar day (not the workday) on which the pouring would begin. He therefore appropriately received no credit for his answer. Question 9 of the Contract Administration Examination was a clear and unambiguous multiple-choice question that fairly tested the candidate's knowledge of the requirements of Section 489.113(3), Florida Statutes, which provides as follows: A contractor shall subcontract all electrical, mechanical, plumbing, roofing, sheet metal, swimming pool, and air- conditioning work, unless such contractor holds a state certificate or registration in the respective trade category, however: A general, building, or residential contractor, except as otherwise provided in this part, shall be responsible for any construction or alteration of a structural component of a building or structure, and any certified general contractor or certified underground utility and excavation contractor may perform clearing and grubbing, grading, excavation, and other site work for any construction project in the state. Any certified building contractor or certified residential contractor may perform clearing and grubbing, grading, excavation, and other site work for any construction project in this state, limited to the lot on which any specific building is located. A general, building, or residential contractor shall not be required to subcontract the installation, or repair made under warranty, of wood shingles, wood shakes, or asphalt or fiberglass shingle roofing materials on a new building of his or her own construction. A general contractor shall not be required to subcontract structural swimming pool work. A general contractor, on new site development work, site redevelopment work, mobile home parks, and commercial properties, shall not be required to subcontract the construction of the main sanitary sewer collection system, the storm collection system, and the water distribution system, not including the continuation of utility lines from the mains to the buildings. A general contractor shall not be required to subcontract the continuation of utility lines from the mains in mobile home parks, and such continuations are to be considered a part of the main sewer collection and main water distribution systems. A solar contractor shall not be required to subcontract minor, as defined by board rule, electrical, mechanical, plumbing, or roofing work so long as that work is within the scope of the license held by the solar contractor and where such work exclusively pertains to the installation of residential solar energy equipment as defined by rules of the board adopted in conjunction with the Electrical Contracting Licensing Board. No general, building, or residential contractor certified after 1973 shall act as, hold himself or herself out to be, or advertise himself or herself to be a roofing contractor unless he or she is certified or registered as a roofing contractor. There was only one correct answer to this question. Approximately 65 percent of the candidates chose this correct response. Petitioner chose another answer that was clearly incorrect inasmuch as a newly licensed general contractor is not free, pursuant to Section 489.113(3)(b), Florida Statutes, to install or repair wood shake roofs on existing buildings constructed by other contractors. Petitioner therefore appropriately received no credit for his answer. Question 29 of the Contract Administration Examination was a clear and unambiguous multiple-choice question that fairly tested the candidate's ability to calculate, based upon the information given, the cost of delivering 28,000 lineal feet of #5 bars of reinforcing steel. There was only one correct answer to this question . Approximately 67 percent of the candidates chose this correct response. Petitioner chose another answer that was clearly incorrect. He therefore appropriately received no credit for his answer. Question 38 of the Contract Administration Examination was a clear and unambiguous multiple-choice question that fairly tested the candidate's ability to distinguish between unit price contracts and other types of contracts, including lump sum contracts. Approximately 82 percent of the candidates chose this correct response. Petitioner chose another answer that was clearly incorrect. He therefore appropriately received no credit for his answer.
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is RECOMMENDED that a final order be entered rejecting Petitioner's challenge to the failing score he received on the contract administration portion of the October 2000 certification examination for general contractors. DONE AND ENTERED this 11th day of May, 2001, in Tallahassee, Leon County, Florida. STUART M. LERNER Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 SUNCOM 278-9675 Fax Filing (850) 921-6847 www.doah.state.fl.us Filed with the Clerk of the Division of Administrative Hearings this 11th day of May, 2001. COPIES FURNISHED: Michael Richards 3802 Lakewood Road Lake Worth, Florida 33461 Charles F. Tunnicliff, Esquire Department of Business and Professional Regulation 1940 North Monroe Street Tallahassee, Florida 32399-2202 Kathleen O'Dowd, Executive Director Construction Industry Licensing Board Department of Business and Professional Regulation 7960 Arlington Expressway, Suite 300 Jacksonville, Florida 32211-7467 Hardy L. Roberts III, General Counsel Department of Business and Professional Regulation Northwood Centre 1940 North Monroe Street Tallahassee, Florida 32399-2202
The Issue The issue to be resolved is whether Petitioner is entitled to additional credit for her response to question nos. 122 and 222 of the civil engineering examination administered on October 31, 1997.
Findings Of Fact On October 31, 1997, Petitioner took the civil professional engineering licensing examination. A score of 70 is required to pass the test. Petitioner obtained a score of 69. Petitioner challenged the scoring of question nos. 122 and 222. As part of the examination challenge process, Petitioner's examination was returned to the National Council of Examiners for Engineering and Surveying where it was re-scored. In the re-score process, the grader deducted points from Petitioner's original score. Petitioner was given the same raw score of 6 on question number 122; however, on question number 222 her raw score of 4 was reduced to a 2. Petitioner needed a raw score of 48 in order to achieve a passing score of 70; she needed at least three additional raw score points to obtain a passing raw score of 48. Petitioner is entitled to a score of 6 on problem number 122. The solution and scoring plan for that problem required the candidate to obtain a culvert size in the range of 21-36 inches. The Petitioner incorrectly answered 3.1 feet or 37.2 inches. She is not entitled to additional credit for problem number 122 because she answered the question with the wrong size culvert. Problem number 122 required the candidate to use a predevelopment peak flow of 40 cubic feet per second (cfs). Petitioner used 58.33 cfs. She chose the maximum flow rather than the predevelopment peak flow. In solving problem number 122, Petitioner chose a design headwater depth of 4.8 feet. The correct solution required a design headwater depth of 5.7 feet. Petitioner made another mistake in problem number 122; she failed to check the water depth in the downstream swale. Petitioner concedes she was given sufficient information to solve problem number 122. She understood what the question was asking of her. She admits that she did not compute the critical depth of the water and that she did not complete the solution. Question number 222 had three parts. The candidate was required to determine the footing size, to select the reinforcing steel, and to provide a sketch for a concrete column located along the edge of a building. Petitioner understood the question and was provided enough information to solve the problem. Petitioner correctly checked the footing size as required by the first part; however, she did not select the reinforcing steel or show the required sketch. Therefore, Petitioner did not complete enough of the problem to qualify for a score of 4 points. She is entitled to a score of 2 points. The examination questions at issue here were properly designed to test the candidate's competency in solving typical problems in real life. The grader (re-scorer) utilized the scoring plan correctly. Petitioner has been in the United States for approximately eleven years. She lived in Romania before she came to the United States. In Romania, Petitioner used only the metric system in her professional work. While she has used the English system since moving to the United States, Petitioner is more familiar with the metric system. The Principles and Practice examination is an open-book examination. Petitioner took a book entitled the Fundamentals of Engineering Reference Handbook to the examination. When the proctor examined her books, she told the Petitioner she was not permitted to keep the handbook. The proctor took the handbook from the Petitioner. Petitioner protested the confiscation of her reference book because she had used the same book in two previous tests. About ten minutes later, the proctor's supervisor returned the book to Petitioner. Petitioner's book was returned at least ten minutes before the test began. She was permitted to use the book during the test. There is no persuasive evidence that the proctor's mistake in temporarily removing Petitioner's reference book caused her to be so upset that she failed the test. Candidates were not permitted to study their books prior to the beginning of the examination. Petitioner may have been nervous when the test began. However, Petitioner received a perfect score of ten points on the first problem she worked, problem number 121.
Recommendation Based upon the findings of fact and conclusions of law, it is RECOMMENDED that the Board of Professional Engineers enter a Final Order confirming Petitioner's score on the examination and dismissing the Petitioner's challenge. DONE AND ENTERED this 13th day of January, 1999, in Tallahassee, Leon County, Florida. SUZANNE F. HOOD Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 SUNCOM 278-9675 Fax Filing (850) 921-6847 www.doah.state.fl.us Filed with the Clerk of the Division of Administrative Hearings this 13th day of January, 1999. COPIES FURNISHED: Natalie A. Lowe, Esquire Board of Professional Engineers 1208 Hays Street Tallahassee, Florida 32301 William Bruce Muench, Esquire 438 East Monroe Street Jacksonville, Florida 32202 Lynda L. Goodgame, General Counsel Department of Business and Professional Regulation 1940 North Monroe Street Tallahassee, Florida 32399-0792 Dennis Bartin, President Florida Engineers Management Corporation 1940 North Monroe Street Tallahassee, Florida 32399-0792
The Issue Whether Petitioner waived its right to protest the Supplemental Notice of Intent to Award issued by the Department on December 1, 2020.
Findings Of Fact On March 17, 2020, the Department issued Request for Proposals Number 06-80101500-J (the RFP) seeking vendors to provide services through state term contracts in two categories (Service Categories): (1) management consulting services (MCS) and (2) financial and performance audit services (FPA). The awarding of state term contracts resulting from the RFP does not guarantee the awarded vendors business; instead, being selected for award under a state term contract merely allows the awarded vendors to further compete for business from state agencies and certain defined eligible users who require the services offered under the contract. After vendors are selected for a state term contract, a state agency or eligible user who requires the services issues a request for quotes from the state term contract vendors. The vendors decide whether they want to compete for the specific services solicited by submitting a quote, and the procuring agency or user then selects from the contracted vendors the vendor that can best fit its unique needs based on the quotes. Accordingly, the purpose of the RFP is simply to pre-qualify the vendors for the future possibility of obtaining work from state agencies and eligible users. Consequently, the Department’s award to multiple vendors for each Service increases competition and gives the users significant choice in selecting a vendor, with each additional award having the effect of increasing the competition and choice available to state agencies and other eligible users who utilize the state term contract. The Department separately evaluated proposals submitted in the two Service Categories and made separate awards for each Service Category. Integrity Group submitted proposals for both Service Categories. Petitioner’s Protest concerns only the actions of the Department in conducting the procurement for the MCS Service Category and does not implicate the FPA Service Category. As part of the evaluation for the MCS Service Category, each vendor submitted a summary of its experience and a separate proposal for each individual Service (Services a through l) within the MCS Service Category. Integrity Group submitted a response summarizing its experience and an individual proposal for each of the MCS Services (Services a through l). Five evaluators appointed by the Department were tasked with scoring each vendor’s response with respect to experience, as well as separately evaluating and scoring each Service proposal submitted for Services a through l. The vendor with the highest score for each Service was awarded a state term contract for such Service, and the RFP reserved to the Department the right to make additional awards to vendors that scored within 25% of the highest score for each Service. The Department initially posted its Notice of Intent to Award and a list of the vendors that were awarded contracts in each Service on September 29, 2020. While the Department awarded Integrity Group state term contracts for the FPA Service Category, it did not make any awards to Integrity Group for the MCS Service Category. The Notice of Intent to Award for the MCS category posted by the Department on September 29, 2020, stated: State of Florida Notice of Intent to Award Management Consulting Services RFP No: 06-801 01500-J Date: September 29, 2020 As to the Management Consulting Services (MCS) category of the above-mentioned Request for Proposals, pursuant to sections 287.057(1)(b) and 120.57(3), Florida Statutes, the Department of Management Services hereby posts its Notice of Intent to Award a contract to the vendors listed in the MCS Award List attachment. Vendors who submitted proposals but were not awarded a Contract are listed in the MCS No Award attachment. Vendors who have submitted proposals deemed non-responsive are listed in the MCS Non-responsive attachment. Failure to file a protest within the time prescribed in section 120.57(3), Florida Statutes, or failure to post the bond or other security required by law within the time allowed for filing a bond shall constitute a waiver of proceedings under chapter 120, Florida Statutes. Any protest concerning this agency decision or intended decision must be timely filed with the Agency Clerk. Protests may be filed by courier, hand delivery, or U.S. mail at Department of Management Services, Office of the General Counsel, Attention: Agency Clerk, 4050 Esplanade Way, Suite 160, Tallahassee, FL 32399-0950. Protests may also be filed by fax at 850-922-6312 or by email at agencyclerk@dms.fl.gov. It is the filing party’s responsibility to meet all filing deadlines. From October 9 to 12, 2020, the Department received formal protests from four vendors not initially selected for award: Intervenor, MGT; TEK Systems Global Services, LLC; Slalom, LLC; and Tidal Basin Government Consulting, LLC. Integrity Group did not file a notice of protest within 72 hours of the Department’s posting of its September 29, 2020, Notice of Intent to Award, and did not file a formal written protest within ten calendar days from the filing of a notice of protest. On December 1, 2020, and after having engaged in resolution conferences with each of the protesting vendors, the Department issued a Supplemental Notice of Intent to Award, which awarded contracts to the four protesting vendors. Thereafter, on December 4, 2020, Integrity Group filed a notice of intent to protest related to the Department’s Supplemental Notice of Intent to Award, and on December 14, 2020, filed Petitioner’s Protest. Petitioner’s Protest alleges that “the Integrity Group is substantially and adversely affected by the Department’s improper and fundamentally flawed procurement process and erroneous decision to exclude the Integrity Group from receiving any awards.” However, as explained in the Conclusions of Law below, Petitioner’s Protest is untimely, has been waived, and should be dismissed.
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is RECOMMENDED that a final order be entered finding that Petitioner lacks standing and dismissing Petitioner’s Protest with prejudice. DONE AND ENTERED this 22nd day of January, 2021, in Tallahassee, Leon County, Florida. S JAMES H. PETERSON, III Administrative Law Judge 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 www.doah.state.fl.us Filed with the Clerk of the Division of Administrative Hearings this 22nd day of January, 2021. COPIES FURNISHED: Rebekah Davis, Esquire Department of Management Services 4050 Esplanade Way Tallahassee, Florida 32399 Marion Drew Parker, Esquire Radey Law Firm Suite 200 301 South Bronough Street Tallahassee, Florida 32301 Mia L. McKown, Esquire Holland & Knight LLP Suite 600 315 South Calhoun Street Tallahassee, Florida 32301 Benjamin J. Grossman, Esquire Foley & Lardner LLP Suite 900 106 East College Avenue Tallahassee, Florida 32301 Christopher Brian Lunny, Esquire Radey Law Firm Suite 200 301 South Bronough Street Tallahassee, Florida 32301 Karen D. Walker, Esquire Holland & Knight, LLP Suite 600 315 South Calhoun Street Tallahassee, Florida 32301 Mallory Neumann, Esquire Foley & Lardner LLP Suite 900 106 East College Avenue Tallahassee, Florida 32301 William D. Hall, Esquire Dean Mead & Hall Suite 1200 106 East College Avenue Tallahassee, Florida 32301 James A. McKee, Esquire Foley & Lardner LLP Suite 900 106 East College Avenue Tallahassee, Florida 32301 William Chorba, General Counsel Office of the General Counsel Department of Management Services 4050 Esplanade Way, Suite 160 Tallahassee, Florida 32399-0950 Jonathan Satter, Secretary Department of Management Services 4050 Esplanade Way, Suite 285 Tallahassee, Florida 32399-0950 Daniel R. Russell, Esquire Dean Mead & Hall Suite 1200 106 East College Avenue, Tallahassee, Florida 32301
The Issue What is the amount of reasonable attorney's fees and costs incurred by Steven B. Feren and the City of Sunrise in connection with the appeal in The Fourth District Court of Appeal Case Number 94-03008, including services rendered in connection with motions for rehearing.
Findings Of Fact On October 18, 1995, The District Court of Appeal, Fourth District, entered an order granting Petitioners, City of Sunrise and Steven B. Feren's, motion for attorney's fees in Case No. 94-03008. A scrivener's error in the order was corrected by order dated November 9, 1995. The corrected order stated: ORDERED that the motion for attorney's fees filed by Stuart Michelson, counsel for appellees, is hereby granted, and pursuant to Fla. R. App. P. 9.400(b), upon remand of this cause the amount thereof shall be assessed by the trial court upon due notice and hearing, subject to review by this court under Fla. R. App. P. 9.400(c). If a motion for rehearing is filed in this court, then services rendered in connection therewith, including but not limited to preparation of a responsive pleading, shall be taken into account in computing the amount of the fee; . . . . Attorney Samuel Goren (Goren) was retained by the Petitioners to testify at the final hearing on the reasonable amount of attorney's fees incurred by Petitioners. Without objection, Goren was qualified and accepted as an expert on the subject of determining reasonable attorney's fees. Prior to rendering an opinion on the reasonable amount of attorney's fees incurred by Petitioners, Goren reviewed the following documents relating to these proceedings: two volumes of pleadings, Stuart Michelson's correspondence file, unfinished drafts of Stuart Michelson's brief, and eight statements from Stuart Michelson (Michelson) for legal services rendered. Goren is familiar with the case of Florida Patient's Compensation Fund v. Rowe, 472 So.2d 1145 (Fla. 1985) and Florida Bar Rule 4-1.5. Goren also interviewed Michelson regarding the work he performed. Based on the foregoing, Goren rendered an opinion on the reasonable hourly rate and the reasonable numbers of hours for the work performed and, consequently, a reasonable legal fee for the services rendered. Michelson charged an hourly rate of $125.00 for his services and $40.00 for the services of paralegals. Goren opined that based on Michelson's skill, experience and reputation, that the hourly rates for attorney's services were very reasonable for the community. Additionally, Goren opined that the rate of $40.00 per hour for paralegal services was also reasonable for the community. I hereby find that the hourly rate of $125.00 for Michelson's services and the hourly rate of $40.00 for the services of a paralegal are reasonable. The services of the Michelson law firm relating to the appeal to the Fourth District Court of Appeal and the motion for rehearing were performed from October 17, 1994, through November 13, 1995. The reasonable amount of attorney hours for the appeal of the final order of the Ethics Commission to the Fourth District Court of Appeal, including services in connection with Colon's motion for rehearing is 88.2. At $125.00 per hour, this amounts to a reasonable fee of $11,025.00. The reasonable amount of paralegal hours for the appeal of the final order of the Ethics Commission to the Fourth District Court of Appeal, including services in connection with Colon's motion for rehearing is 15.45. At $40.00 per hour, this amounts to a reasonable fee of $618.00. The services of the Michelson law firm relating to the determination of the amount of the fees was performed from December 18, 1995 through the date of the final hearing. In addition there will be additional time which will be necessary to bring the proceeding to a Final Order. Michelson submitted invoices to Petitioners dated February 9, 1996 and May 16, 1996. A review of the invoices indicates that some of the services provided did not relate to the fee hearing but to other matters including a lien. Based on the invoices the following dates appear to be for services related to the fee hearing: December 18, 19, 21, and 30, 1995; January 10, 18, 19, and 25, 1996; May 6, 7, 10, 11, 14, 15, and 16, 1996; and May 8 and 9, 1996, excluding lien services. Based on these invoices the reasonable amount of attorney hours for the determination of the amount of fees through May 16, 1996 was 12.5, which equates to $1,562.50. Goren opined that an additional 20 to 25 attorney hours will be needed to bring this case to final conclusion with a final order being entered by the Ethics Commission. A reasonable number of attorney hours necessary to bring this case to final conclusion from the date of the final hearing to the entry of a final order by the Ethics Commission is 25 hours and a reasonable rate for the attorney's time is $125 per hour. The reasonable amount of fees that would be incurred from the final hearing to the entry of a final order by the Ethics Commission is $3,125.00 The attorney and paralegal services and costs contained in the December 8, 1995, statement from the Michelson law firm deal with a petition for certiorari and are not within the scope of the order by the Fourth District Court of Appeal. The amount of reasonable costs incurred by Petitioners in this proceeding and the appeal to the Fourth District Court of Appeal is $896.70. Goren charged the Petitioners $350 for services as an expert witness in connection with these proceedings from October, 1995 through April 30, 1996. Goren spent an additional 9.7 hours on this matter through the final hearing. Goren's hourly rate of $125 is reasonable. The number of hours spent by Goren is reasonable. Goren's fee of $1,562.50 is a reasonable fee for an expert witness in this proceeding.
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is RECOMMENDED a final order be entered awarding Petitioners, Steven B. Feren and the City of Sunrise $18,789.70 for reasonable attorney's fees and costs. DONE AND ENTERED this 29th day of May, 1996, in Tallahassee, Leon County, Florida. SUSAN B. KIRKLAND, Hearing Officer Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-1550 (904) 488-9675 Filed with the Clerk of the Division of Administrative Hearings this 29th day of May, 1996. APPENDIX TO RECOMMENDED ORDER, CASE NO. 95-6181FE To comply with the requirements of Section 120.59(2), Florida Statutes (1995), the following rulings are made on the parties' proposed findings of fact: Petitioner's Proposed Findings of Fact. Paragraphs 1-3: Accepted. Paragraphs 4-6: Rejected as not supported by the evidence. Paragraphs 7-8: Accepted. Respondent's Proposed Findings of Fact. The Respondent's Recommended Order did not contain findings of fact. COPIES FURNISHED: Stuart R. Michelson, Esquire 1111 Kane Concourse, Suite 517 Bay Harbor Islands, Florida 33154 Mr. Bill Colon 11640 Northwest 30th Place Sunrise, Florida 33323 Carrie Stillman, Complaint Coordinator Commission on Ethics Post Office Box 15709 Tallahassee, Florida 32317-5709 Bonnie Williams, Executive Director Florida Commission On Ethics Post Office Drawer 15709 Tallahassee, Florida 32317-5709 Phil Claypool, General Counsel Ethics Commission 2822 Remington Green Circle, Suite 101 Post Office Drawer 15709 Tallahassee, Florida 32317-5709
