The Issue The issue for determination is whether Petitioner’s challenge to the failing score he received on the essay portion of the Florida Teacher Certification Examination’s (FTCE) General Knowledge (GK) test should be sustained.
Findings Of Fact Petitioner has been employed as a teacher for the past three years. He had a temporary Florida teacher certificate, but at the time of the hearing, he said that he believes it was expired. Petitioner is seeking to qualify for a (non-temporary) Florida teacher certificate. Petitioner first must pass the essay part of the GK test to complete the GK requirements. He would then be qualified to proceed to address the remaining certification requirements. See § 1012.56(2)(g), (h), (i), and (7), Fla. Stat. Respondent, Pam Stewart, as Commissioner of Education, is the state’s chief educational officer and executive director of DOE. §§ 20.15(2) and 1001.10(1), Fla. Stat. One of DOE’s responsibilities is to review applications for educator certification and determine the qualifications of applicants according to eligibility standards and prerequisites for the specific type of certification sought. See § 1012.56, Fla. Stat. One common prerequisite is taking and passing examinations relevant to the particular certification. Respondent is authorized to contract for development, administration, and scoring of educator certification exams. § 1012.56(9)(a), Fla. Stat. Pursuant to this authority, following a competitive procurement in 2011, Pearson was awarded a contract to administer and score Florida’s educator certification exams, including the FTCE. The State Board of Education (SBE) is the collegial agency head of DOE. § 20.15(1), Fla. Stat. As agency head, the SBE was required to approve the contract with Pearson. The SBE is also charged with promulgating certain rules that set forth policies related to educator certification, such as requirements to achieve a passing score on certification exams. DOE develops recommendations for the SBE regarding promulgating and amending these rules. In developing its recommendations, DOE obtains input and information from a diverse group of Florida experts and stakeholders, including active teachers, school district personnel, and academicians from colleges and universities. FTCE Development, Administration, and Scoring DOE develops the FTCE, as well as the other educator certification exams, in-house. The FTCE is developed and periodically revised to align with SBE-promulgated standards for teachers. In addition, as required by statute, certification exams, including the FTCE, must be aligned to SBE-approved student standards. Details about the FTCE, such as the competencies and skills to be tested, the exam organization, and passing score requirements, are set forth in Florida Administrative Code Rule 6A-4.0021 (the FTCE rule). The FTCE rule has been amended periodically, but the current version includes a running history, setting forth FTCE details that applied during past time periods, as well as those currently in effect. The FTCE is not actually a single examination. It consists of multiple separate examinations to meet the different requirements for teacher certification and the different options for specific subject areas. Descriptions of the areas to be tested by each FTCE component are set forth in a publication incorporated by reference in the FTCE rule. The version of this publication that was in effect when Petitioner took the exam at issue in this proceeding is identified in the FTCE rule as: “Competencies and Skills Required for Teacher Certification in Florida, Twenty-Second Edition.” As set forth in the FTCE rule, the GK exam consists of four subtests. Subtest one is the essay test; subtest two, three, and four are multiple choice tests covering English language skills, reading, and math, respectively. Petitioner met the requirements for GK subtests two, three, and four, by virtue of having taken and passed the College Level Academic Skills Test (CLAST) in those areas prior to July 1, 2002.3/ Therefore, Petitioner only had to take and pass subtest one, the essay exam, to satisfy all GK requirements. The competency and skills to be tested by the GK essay test, as promulgated by the SBE and codified by reference in the FTCE rule, are as follows: Knowledge of formal college-level writing Determine the purpose of writing to task and audience. Provide a section that effectively introduces the topic. Formulate a relevant thesis or claim. Organize ideas and details effectively. Provide adequate, relevant support by citing ample textual evidence; response may also include anecdotal experience for added support. Use a variety of transitional devices effectively throughout and within a written text. Demonstrate proficient use of college- level, standard written English (e.g., varied word choice, syntax, language conventions, semantics). Provide a concluding statement or section that follows from, or supports, the argument or information presented. Use a variety of sentence patterns effectively. Maintain a consistent point of view. Apply the conventions of standard English (e.g., avoid inappropriate use of slang, jargon, clichés). (Competencies and Skills Required for Teacher Certification in Florida, Twenty-Second Edition, page 2 of 247, incorporated by reference in the FTCE rule). Prior to January 1, 2015, a score of at least six (using a scoring range from two points to 12 points) was required to pass the GK essay test. Based on input from educators, academicians, and other subject matter experts, DOE recommended that the passing score for the GK essay test be raised from a score of six to a score of eight (using the same range of two points to 12 points). The SBE adopted the recommendation, which is codified in the FTCE rule: eight is the required passing score for GK essays as of January 1, 2015. Without question, the higher passing score requirement makes it more difficult to pass the GK essay. The policy underlying this scoring change is to make the GK essay test more rigorous, in recognition of the critical importance of writing skills. By raising the standards for demonstrating mastery of the writing skills tested by the GK essay test, the GK essay test better aligns with increasingly rigorous SBE-approved student standards for written performance. This policy change is reasonable and within the purview of the SBE; in any event, it is not subject to debate in this case, because Petitioner did not challenge the FTCE rule. Not surprisingly, since the passing score was raised for the GK essay, the overall passage rates have dropped. The passage rates were 96 percent in 2013 and 93 percent in 2014, when the passing score was lower. After the passing score was raised, the passage rates were 63 percent in 2015 and 69 percent in 2016. While Petitioner characterizes the 69 percent passage rate as “low” (Pet. PRO at 4, ¶ 13), that is an opinion that is unsupported by any testimony offered at hearing. Petitioner did not offer any expert witness to testify on his behalf. Instead, based on the testimony offered on this subject at the final hearing, the more reasonable inference to draw from the overall GK essay passage rates is that the passage rates were too high prior to 2015. The overall GK essay passage rate, standing alone, is not evidence that the GK essay is arbitrary, capricious, unfair, or invalid. Pursuant to its contract with DOE as the test administration and test scoring agency, Pearson administers and scores GK essay exams. Pearson employs holistic scoring as the exclusive method for scoring essays, including GK essays (as specified in Pearson’s contract with DOE). The holistic scoring method is used to score essay examinations by professionals across the testing service industry. Pearson has extensive experience in the testing service industry, currently providing test scoring services to more than 20 states. Dr. Michael Grogan, Pearson’s director of performance assessment scoring services and a former chief rater, has been leading sessions in holistic scoring or training others since 2003. He described the holistic scoring method as a process of evaluating the overall effect of a response, weighing its strengths and weaknesses, and assigning the response one score. Through training and use of tools, such as rubrics and exemplars, the evaluation process becomes less subjective and more standardized, with professional bias of individual raters minimized, and leading to consistent scoring among trained raters. Training is therefore an integral part of Pearson’s testing services for which DOE contracted. In an intensive two- day training program, prospective raters are trained in the holistic scoring method used to score GK essays. Pearson’s rater training program begins with a review of background about the holistic scoring method generally, including discussions about rater bias. From there, trainees are oriented to GK essay-specific training material. They thoroughly review and discuss the rubric, the score scale (which is one point to six points), the operational prompt raters will be scoring, and exemplars (other responses to the prompt that have been pre-scored). The rater candidates then employ these tools to begin independently scoring exemplars. Raters-in-training conduct many rounds of independent scoring sessions, interspersed with group discussions regarding how the essays should have been scored. The trainees then move into the calibration test phase, in which they independently score essay exemplars, paired with an experienced rater who independently scores the same exemplars. The trainees score essay after essay, then compare scores with the experienced rater, with the goal to achieve consistency in scores, by equaling or coming within one point of the other rater’s score. Ultimately, the raters must pass the calibration test by achieving scoring consistency to qualify for appointment as raters to score actual GK essays. Raters who conduct scoring of the GK essay must meet qualifications specified by DOE (including teacher certification and experience). Pearson proposes qualified individuals to DOE, and then DOE must approve proposed raters. Then the approved raters must undergo and successfully complete Pearson’s training. Each GK essay is scored independently by two qualified raters. Pairs of raters receive scoring assignments, one prompt at a time. The assignments are received anonymously; one rater does not know who the other assigned rater is. And neither rater knows anything about the examinee, as the essay is identified solely by a blind number. GK essay raters work in one room, at individual computer terminals, in Hadley. Security of all testing information is vigilantly maintained, through confidentiality agreements and secure, limited, and protected computer access. For each scoring assignment, raters adhere to a step- by-step process that reinforces their initial training. Raters must first score sample responses to a historic prompt that is different from the assigned prompt, as a training refresher to invoke the holistic scoring mindset. From there, raters review the assigned prompt and the scoring rubric. Raters then must score an anchor set of six sample responses, one exemplifying each score category; the historic scores are not revealed until the raters complete their scoring. Raters compare their scores with the anchor scores and work through any discrepancies. Raters then go through a calibration process of scoring 10 more sample responses to the same prompt. After scoring all 10 essays, the raters learn the scores deemed appropriate for those responses, and must work through any discrepancies until consistency is achieved. Only after scoring many sample essays and achieving scoring consistency are the raters permitted to turn to the assigned GK essay for review and scoring. Pearson also employs chief raters to supervise and monitor the raters while they are engaged in their scoring work. Chief raters must meet specified qualifications and be approved by DOE. Chief raters must be certified and experienced in the field of teaching, plus they must have prior experience working as raters. Chief raters conduct the training sessions to train raters in the holistic scoring method in Hadley. A chief rater supervises and monitors raters by being physically present in the same room with the raters while they are engaged in their scoring work. The chief rater monitors rater work online in real time. As raters enter scores, those scores are immediately known by the chief rater, so that any “red flag” issues in scoring results and trends can be addressed immediately. The scores of the two raters assigned to score a GK essay are added together for the total holistic score. Thus, the total score range for a GK essay is between two points and 12 points: the lowest possible score of two points would be achieved if each rater assigns a score of one point; and the highest score of 12 points would be achieved if each rater assigns six points. The sum of the two raters’ scores will be the score that the GK essay receives unless the raters’ scores disagree by more than one point. If the two raters’ scores differ by more than one point, then the chief rater steps in to resolve the discrepancy. After GK essays are scored, the examinee is informed of the final score of between two and 12 points, and the examinee is told whether the score is a passing or failing score. Eight points is a passing score, according to the FTCE rule. Raters do not develop written comments as part of their evaluation of GK essays. Their holistic evaluation is expressed by the point value they assign to the essay. Through the intensive training and the subsequent calibration and recalibration before each GK essay scoring assignment, Pearson has achieved consistency in rater scoring of GK essays that meets industry standards for holistic scoring. Consistency in this context means that the scores assigned to a GK essay by a pair of raters are either identical or adjacent (within one point), and when adjacent, are balanced (i.e., each rater is as often the higher scorer as he or she is the lower scorer). DOE makes sure that Pearson maintains rater scoring consistency in accordance with industry standards, by monitoring monthly performance reports provided by Pearson. Examinee Perspective: Preparation for the GK Essay DOE provides detailed information and aids on its website regarding all four subtests of the GK exam, including the GK essay, for potential examinees. This includes a 39-page test information guide for the FTCE GK test. The test information guide sets forth the complete SBE- adopted competencies and skills to be tested by each of the four GK subtests, including those specific to the essay test quoted in Finding of Fact 11. The test information guide explains the GK essay and scoring process, as follows: For your essay, you will choose between two topics. The 50 minutes allotted for this section of the exam includes time to prepare, write, and edit your essay. Your work will be scored holistically by two raters. The personal views you express will not be an issue; however, the skill with which you express those views, the logic of your arguments, and the degree to which you support your position will be very important in the scoring. Your essay will be scored on both the substance and the composition skills demonstrated, including the following elements: ideas, organization, style (diction and sentence structure), and mechanics (capitalization, punctuation, spelling, and usage). The raters will use the categories on page 14 when evaluating your essay. The score you receive for your essay will be the combined total of the two raters’ scores. (R. Exh. 2 at 12 of 39). At the referenced page 14, the test information guide sets forth in full the scoring rubric used by raters to evaluate GK essays. The rubric is simply a comparative description of the extent to which an essay demonstrates the competency and skills to be tested, on a scoring scale of one to six points. The rubric descriptions differentiate between the various skills to be tested in a way that identifies, as to each skill or group of skills, which essay is best, better, good, not-so-good, worse, and worst. But the evaluation of each skill is not separately scored; instead, the essay response is evaluated as a whole, with the various strengths and weaknesses weighed and balanced. Finally, the test information guide provides a sample essay test, with representative essay prompts in the same format that the examinee will see on the exam: two topics are set forth, with instructions that the examinee is to select one of the two topics. The information DOE makes publicly available is appropriate and sufficient to explain the GK essay exam and scoring process, and to allow an examinee to know what to expect in a prompt and what is expected of the examinee in a response. Score Verification An examinee who fails the GK essay test (or any other FTCE test or subtest) may request score verification to verify that the failed exam was scored correctly. The examinee has the right, by statute and rule, to review the test question(s) and response(s) that received a failing score. The score verification procedures, providing this review opportunity, are set forth in the FTCE rule. The score verification rule provides that DOE makes the determination as to whether an examinee’s test was scored correctly. DOE is authorized to consult with field-specific subject matter experts in making this determination. In practice, though not required by the FTCE rule, when a score verification request is directed to the scoring of a GK essay, DOE always consults with a field-specific subject matter expert known as a “chief reviewer.” Chief reviewers are another category of experts (in addition to raters and chief raters) retained by Pearson, pursuant to qualifications identified by DOE, and subject to DOE approval. Once approved by DOE, prospective chief reviewers undergo the same rater training in the holistic scoring process as do all other raters, to gain experience in scoring essays and undergo calibration to achieve scoring consistency. In addition, chief reviewers are given training for the chief reviewer role of conducting review and scoring of essays when scores have been contested. Unlike raters and chief raters, chief reviewers do not work at Pearson in Hadley; they are Florida experts in the field, with certification and experience teaching in Florida schools. Chief reviewers only become involved with GK essays when an examinee who failed the GK essay invokes the score verification process. A chief reviewer is assigned to evaluate whether that essay was scored correctly. As with the initial scoring, a chief reviewer is not given any information about the raters or about the examinee; the essay is assigned a blind, anonymous number. The chief reviewer conducts the evaluation by first going through the same step-by-step process as raters, following the same retraining and calibration steps that involve scoring many sample essays. Upon achieving success in the calibration test, the chief reviewer moves on to evaluate the assigned essay response independently, before reviewing the scores the raters gave to that essay. After reviewing the raters’ scores, the chief reviewer offers his or her view as to whether the essay score should stand or be changed, and provides a summary rationale for that opinion. This information is conveyed to DOE, which determines the action to take--verify or change the score--and notifies the examinee of the action taken. In the 14-month period from January 2016 through February 2017, two failing GK essay scores were changed by DOE to passing scores as a result of the score verification process. As with the overall passage rates, Petitioner characterizes this reversal rate as low, but no evidence is offered to prove that characterization. It is as reasonable or more reasonable to infer from the fact that GK essay scores are only rarely reversed through score verification that the scoring process works well. Petitioner’s GK Essay Attempts Petitioner took the GK essay test for the first time in July 2015. He received a failing score of four, with two points assigned by each of the two raters. Petitioner admits that he did little to nothing to prepare for the GK essay the first time. When taking the essay test, he ran out of time and recalls that he left the essay incomplete. The time pressure “had a huge deal with me not being able to provide enough specifics for it to make any sense at all where I was going with the essay.” (Tr. 75). Petitioner thought the passing score was six at the time, but his recollection is incorrect. The higher passing score of eight has been in place since January 2015, and has been the passing score for each of Petitioner’s GK essay attempts. FTCE examinees can retake failed subtests/sections, and need only retake the parts failed. There are no limits on the number of retakes. The requirements for retakes are that at least 30 days must have elapsed since the last exam attempt, and that examinees pay the registration fees specified in the FTCE rule for each retake of a failed subtest and/or section. Petitioner retook the GK essay test in February 2016. In preparation for this second attempt, Petitioner did not seek tutoring or spend much time training. As he explained, “I’m under the impression that I can write an essay.” (Tr. 21). Instead, he focused mostly on preparing for the timed aspect of the exam, making sure that he started when the clock started. Although his score improved from four to six, it was still a failing score. Petitioner did not invoke the score verification process to question the failing scores he received on his first two GK essays. Those two failing scores stand as final, as he did not challenge them. Petitioner took the GK essay test for the third time on June 25, 2016. This time, he prepared to some extent. In the month before the exam, Petitioner sought help from someone he described as a writing coach. The writing coach did not evaluate Petitioner’s writing so as to identify weaknesses; instead, she asked him what he thought his weaknesses were, and he responded that he did not know what his weakness is besides not being able to formulate his plan and map out his essay faster. As a result, she coached him on some mapping techniques, and on how to structurally organize an essay--with an introduction, followed by three points in paragraphs begun with transitional phrases, and a conclusion. Petitioner practiced a little with his writing coach, by email: she would send a prompt and he would write an essay, which he timed, and then send it back to her. They did this “a few times.” (Tr. 24). There is no evidence of record regarding the writing coach, other than that her name is Ms. Martin. She may have been Petitioner’s proposed witness who was allowed to appear from New York by telephone, but who was not called to testify. One of the things Petitioner learned from Ms. Martin was that in his introduction, he should “speak vaguely about” what will be covered. When asked if Ms. Martin actually said to be “vague” in the beginning, Petitioner said, “She may not have used the word vague, but that is the meaning that I got from what she said.” (Tr. 70). In preparation for his third attempt at the GK essay test, Petitioner also sought help from Jordan Gibbs, who was described as an educator who taught language arts for over 20 years. Petitioner testified that Mr. Gibbs is “our academy leader there[.]” (Tr. 24). However, Petitioner did not elaborate; it is unknown which academy is led by Mr. Gibbs, or where “there” is. Like Ms. Martin, Mr. Gibbs also addressed mapping techniques with Petitioner. Petitioner never sent any essay drafts to Mr. Gibbs for his review. Petitioner also reviewed GK essay preparation material on the DOE website. He reviewed sample prompts, but did not practice writing complete essays. He just looked at the sample prompts for purposes of mapping and planning an essay. Petitioner said that he found the preparation material useful to an extent, but did not think the sample prompts reflected the type of GK essay prompts in use when he took the test. A comparison of the sample GK essay prompts in the test information guide (R. Exh. 2 at 17 of 39) with the actual GK essay prompt Petitioner chose for his essay topic (Jt. Exh. 1 at 3 of 4) suggests otherwise. Although DOE obviously does not make available as samples the actual essay prompts actively being used in GK examinations, the sample prompts appear to be similar to Petitioner’s actual prompt in style, substance, and tone. It would be unreasonable for examinees to expect more from a testing agency than what DOE makes available. Petitioner’s score improved slightly in his third attempt at the GK essay test, but it was still a failing score of seven. One rater assigned the essay a score of three, while another rater scored the essay a four. Each of the three times Petitioner took the GK essay test, the two raters assigned scores that were consistent, in that they were either identical or adjacent (within one point of each other). Accordingly, a chief rater was never assigned for discrepancy resolution, as there were no discrepancies. After receiving notification of his third failing score, this time Petitioner invoked the score verification process. Petitioner completed a statement explaining why he believes his score was erroneous, which is in evidence as part of the confidential testing material. (Jt. Exh. 1 at 2 of 4). The statement set forth why he believes the essay demonstrated good organization, used transitional phrases, and addressed the topic. He acknowledged one misspelling, and acknowledged that his conclusion ended in mid-sentence, as he ran out of time. He added three words to complete the last sentence, and suggested that the ending should have been inferred from what he did say. DOE conducted its review, and the score was verified through a process consistent with DOE’s practice of consulting a chief reviewer who was qualified as a subject matter expert in the field of teaching in Florida and approved by DOE. The chief reviewer who undertook to verify Petitioner’s essay score conducted an independent evaluation of Petitioner’s essay following the same holistic method. Then the chief reviewer considered the scores separately assigned by the two raters who scored Petitioner’s essay. She concluded that the assigned scores of three/four should stand. The chief reviewer provided a summary rationale for her determination, offering her view that the essay borders on a three/three due to weak development.4/ The chief reviewer’s summary was provided to DOE for consideration. By letter dated September 27, 2016, Petitioner was notified by DOE that the “essay score that you questioned has been reviewed by a Chief Reviewer. As a result of this review, the Department has determined that the written performance section that you questioned is indeed scored correctly.” Petitioner was notified of his right to an administrative hearing pursuant to sections 120.569 and 120.57 to dispute the decision. Petitioner availed himself of that opportunity, and was given the chance in a de novo evidentiary hearing to present evidence to support his challenge to his exam score. At the hearing, Petitioner offered only his own testimony as support for his challenge to the scoring of his essay. Petitioner was not shown to be, tendered as, or qualified as an expert in either formal college-level English writing or scoring of essays. His attempt to compare isolated parts of the rubric to isolated parts of his essay is contrary to the holistic scoring approach used to score the GK essay. Petitioner offered no comprehensive, holistic evaluation of his essay as a whole, nor was he shown to be qualified to do so. Besides being contrary to the holistic scoring method, Petitioner’s critique of the scoring of his essay was wholly unpersuasive. Without undermining the confidentiality of the ingredients of Petitioner’s testimony (the essay prompt, his essay, and the historic anchors), overall, the undersigned did not find Petitioner’s critique credible or accurate. Although awkward to try to explain in code, some examples follow to illustrate the basis for this overall finding. Petitioner began his critique by reading the first three sentences--the introductory paragraph--of his essay. He said that each sentence had one topic, and that each of the subsequent three paragraphs in the body addresses one of those three topics. The problem with Petitioner’s explanation for the substantive organization of his essay is that the essay prompt identifies a single topic, not three topics. Petitioner failed to respond to the prompt’s single topic by introducing that topic as the essay’s theme, and developing that single theme in the body of that essay. Similarly, the concluding paragraph offers scattered thoughts, somewhat related to the three topics discussed in the essay. The essay’s weakness in development was a prominent point in the scoring rationale summaries written by the raters and chief reviewers. Petitioner specifically addressed only one aspect of the rubric considerations, addressing the extent to which an essay has errors in sentence structure, usage, and mechanics. As to this consideration, Petitioner stated that there were three spelling errors in his essay (up from the one error he identified in his score verification statement). He was critical of one rater’s comments for referring to grammatical errors, because Petitioner does not believe there were any grammatical errors in his essay. Petitioner’s assessment of his essay reflects his bias, because it fails under any objective analysis. In fact, Petitioner’s essay (Jt. Exh. 3) has both spelling errors and grammatical errors. In addition, the essay uses poor sentence structure in several instances, as well as poor word choices that interfere with an understanding of what Petitioner means. An example of a sentence with a grammatical error is the fifth sentence in paragraph 4. At the very least, the word “having” is required after the comma. With that addition, the sentence would only be awkward, instead of grammatically incorrect. An example of a poorly written sentence is the second sentence of the second paragraph. This sentence combines a misspelling, a misused word, and syntax that is awkward, at best. Petitioner must also acknowledge that the last sentence of his essay is another example of poor sentence structure, since it is an incomplete sentence without punctuation. It would be inappropriate for raters reviewing essays to fill in the gaps left by writers, whether those gaps were because of running out of time or otherwise. What Petitioner meant to write to complete the sentence is not something that can be added after-the-fact to cure the defect on the face of the essay. By the undersigned’s count, there are five misspellings in the essay, unless one counts “in to,” which should be “into,” as an error of grammar or syntax. The other misspellings were: easire (easier); savy (savvy); yeild (yield); and evironment (environment). In addition, Petitioner made several punctuation errors, failing to hyphenate two compound adjectives preceding nouns and presenting a single idea: cutting-edge technology; tech-savvy students. Petitioner also improperly omitted a hyphen in “self discipline.” Petitioner acknowledged some repetitive use of a particular word, but thought he only used that word twice. In fact, he used the word in both sentences one and two of the second paragraph, and then again in paragraph four. Only the first usage is arguably correct (but in an awkwardly written sentence). While used once, the word is an interesting one, Petitioner’s overuse and misuse of this word suggests a mechanical, as opposed to thoughtful, approach of injecting interesting words into the essay. Petitioner’s essay demonstrated good superficial structure, with an introductory paragraph, three paragraphs in the body that begin with good transitional words, and a concluding paragraph. The organizational structure may have earned Petitioner a score of four, as stated in that rater’s comments, but that same rater also repeated the comments of others that where the essay is weakest is in development. Petitioner offered his view that the only reason his essay received a failing score was because the raters considered it to be too short in length. While Petitioner is correct in noting that length is not a criterion, he mischaracterized the comments on this subject, by ignoring the criticisms of his essay that were made when the length of the essay was noted. The comments only mention the length of Petitioner’s essay as it correlates to other considerations, such as the weakness in development, the lack of specifics or examples, or the impact of a “number of misspellings, . . . usage issues, . . . and punctuation errors,” which accumulated to a notable level “given the shortness of the response.” (Jt. Exh. 5-A). Petitioner failed to prove his contention that an unauthorized criterion-- essay length alone--was applied in scoring Petitioner’s essay. Petitioner failed to prove that the holistic scoring of his essay was incorrect, arbitrary, capricious, or devoid of logic and reason. He offered no evidence that a proper holistic evaluation of his essay would result in a higher total score than seven; indeed, he offered no holistic evaluation of his essay at all. Petitioner’s critique of various parts in isolation did not credibly or effectively prove that his score of seven was too low; if anything, a non-expert’s review of various parts in isolation could suggest that a score of seven would be generous. But that is not the scoring approach called for here. Petitioner presented no evidence that any aspect of the GK essay process overall, including development, administration, evaluation, and score review, was arbitrary, capricious, unfair, discriminatory, or contrary to requirements imposed by law.
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is RECOMMENDED that a final order be entered rejecting Petitioner’s challenge to the failing score he received on the General Knowledge essay test taken in June 2016, and dismissing the petition in this proceeding. DONE AND ENTERED this 13th day of October, 2017, in Tallahassee, Leon County, Florida. S ELIZABETH W. MCARTHUR Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 Fax Filing (850) 921-6847 www.doah.state.fl.us Filed with the Clerk of the Division of Administrative Hearings this 13th day of October, 2017.
The Issue Did Petitioner pass the Mechanical Engineers Examination he took on April 24, 1998?
Findings Of Fact On April 24, 1998, Petitioner took the Mechanical Engineers Examination. He received a score of 69 for his effort. A passing score was 70. The Mechanical Engineers Examination was administered under Respondent's auspices. As alluded to in the preliminary statement, Petitioner challenged the score received on problem 146. The maximum score available for that problem was ten points. Petitioner received eight points. In accordance with the National Council of Examiners for Engineering and Surveying Principles in Practice of Engineering Examinations for spring 1998, score conversion table - discipline specific, Petitioner had a raw score of 47 which equated to a conversion of 69, to include the eight raw points received for problem 146. In addition, the examination provided a scoring plan for problem 146, which assigns scores in increments of two points from zero to ten. To pass, it would be necessary for Petitioner to receive an incremental increase of two points, raising his score from eight points to ten points. This would give him a raw score of 49 points. According to the score conversion table - discipline specific, that would give Petitioner 71 points. According to the scoring plan for problem 146 to receive the ten points, Petitioner would have to demonstrate: Exceptional competence (it is not necessary that the solution to the problem be perfect) generally complete, one math error. Shows in-depth understanding of cooling load calculation psychrometrics. Problem 146 required Petitioner to: Determine the required cooling coil supply air quantity (cfm) and the conditions (°F db and °F wb) of the air entering and leaving the coil." Petitioner was provided a psychrometric chart to assist in solving problem 146. The examination candidates were also allowed to bring reference sources to the examination to assist in solving the examination problems. Petitioner brought to the examination, the Air-Conditioning Systems Design Manual prepared by the ASHRAE 581-RP Project Team, Harold G. Lorsch, Principal Investigator. Petitioner used that manual to determine the wet-bulb temperature of the air entering the coil. In particular, he used an equation from the manual involving air mixtures. For that part of the solution he arrived at a temperature of 65.6°F wb. According to the problem solution by Respondent's affiliate testing agency, reference ASHRAE Fundamentals Chapter 26, the coil entering wet-bulb temperature taken from the psychrometric chart was 66.12°F wb. The scorer in grading Petitioner's solution for problem 146 placed an "x" by the answer provided 65.6°F wb and wrote the words "psychrometric chart." No other entry or comment was made by that scorer in initially reviewing the solution Petitioner provided for that problem. This led to the score of eight. The scoring plan for problem 146 for the April 1998 examination taken by Respondent equates the score of eight as: MORE THAN MINIMUM BUT LESS THAN EXCEPTIONAL COMPETENCE Either a) Provides correct solution to problem with two math errors or incorrect dry-bulb or wet-bulb for coil entering or leaving conditions or minor total cooling load error, or b) Provides correct solution to items c and d correctly and minor math errors in items a and b of Score 6 below. Petitioner was entitled to review the results of his examination. He exercised that opportunity on September 21, 1998, through a post-examination review session. Petitioner requested and was provided re-scoring of his solution to problem 146. According to correspondence from the National Council of Examiners for Engineering and Surveying to the Florida Member Board from Patricia M. Simpson, Assistant Supervisor of scoring services, the score did not change through re-scoring. In this instance, the October 14, 1998 correspondence on re-scoring states, in relation to problem 146: Incorrect methodology used in calculating coil entering wet-bulb temperature. Incorrect coil entering wet-bulb temperature provided. No calculation provided for coil leaving temperature conditions. The coil leaving wet-bulb temperature in Respondent's proposed solution was 53.22°F wb taken from the psychrometric chart. Petitioner's solution for the coil leaving wet-bulb temperature taken from the psychrometric chart was 53.3°F wb. At hearing Respondent did not provide an expert to establish the basis for point deduction in the original score and the re-scoring of Petitioner's solution for problem 146. Moreover, Respondent did not present expert witnesses to defend the commentary, the preferred written solution in its examination materials. Consequently, Respondent's preferred solution constitutes hearsay about which no facts may be found accepting the validity of Respondent's proposed solution, as opposed to merely reporting that information.1 By contrast, Petitioner provided direct evidence concerning the solution provided for problem 146 in response to the criticisms of his solution that were unsupported by competent evidence at hearing. More importantly the criticisms were responded to at hearing by Geoffrey Spencer, P.E., a mechanical engineer licensed to practice in Florida, who was accepted as an expert in that field for purposes of the hearing. As Petitioner explained at hearing, he used the Air- Conditioning Systems Design Manual equation to arrive at the coil entering wet-bulb temperature, which he believed would provide the answer as readily as the use of the psychrometric chart. (Although the psychrometric chart had been provided to Petitioner for solving problem 146, the instructions for that problem did not prohibit the use of the equation or formula.) Petitioner in his testimony pointed out the equivalency of the process of the use of the psychrometric chart and the equation. Petitioner deemed the equation to be more accurate than the psychrometric chart. Petitioner had a concern that if the answer on the coil entering wet-bulb temperature was inaccurate, this would present difficulty in solving the rest of problem 146 because the error would be carried forward. Petitioner pointed out in his testimony that the solution for determining the coil entering wet-bulb temperature was set out in his answer. The answer that was derived by use of the formula was more time consuming but less prone to error, according the Petitioner's testimony. Petitioner points out in his testimony that the answer he derived, 65.6°F wb, is not significantly different than Respondent's proposed solution of 66.12°F wb. (The instructions concerning problem 146 did not explain what decimal point of a degree the candidate had to respond to in order to get full credit for that portion of the solution to the problem.) Petitioner in his testimony concerning his solution for the coil leaving wet-bulb temperature indicated that the calculation for arriving at that temperature was taken from the psychrometric chart and is sufficiently detailed to be understood. Further, Petitioner testified that the degree of accuracy in which the answer was given as 53.3°F wb, as opposed to Respondent's proposed solution of 53.22°F wb, is in recognition of the use of the psychrometric chart. Petitioner questions whether the proposed solution by Respondent, two decimal points, could be arrived at by the use of the psychrometric chart. In relation to the calculation of the coil entering wet-bulb temperature, Mr. Spencer testified that the formula from the Air-Conditioning Systems Design Manual or the psychrometric chart could have been used. Moreover, Mr. Spencer stated his opinion that the solution for coil entering wet-bulb temperature of 65.6°F wb by Petitioner is sufficiently close to Respondent's proposed solution of 66.12°F wb to be acceptable. Mr. Spencer expressed the opinion that Petitioner had correctly used the formula from the manual in solving the coil entering wet-bulb temperature. Mr. Spencer expressed the opinion that the psychrometric chart is an easier source for obtaining the solution than the use of the formula from the manual. In Mr. Spencer's opinion, the formula shows a more basic knowledge of the physics involved than the use of the psychrometric chart would demonstrate. In relation to the coil leaving wet-bulb temperature, Mr. Spencer expressed the opinion that Petitioner had adequately explained the manner of deriving the answer. Further, Mr. Spencer expressed the opinion that the answer derived was sufficiently accurate. The testimony of Petitioner and opinion of Mr. Spencer is unrefuted and accepted.
Recommendation Upon consideration of the facts found and conclusions of law reached, it is RECOMMENDED: That a final order be entered which finds that Petitioner passed the Florida Board of Professional Engineers April 24, 1998, Mechanical Engineers Examination with a score of 71. DONE AND ENTERED this 22nd day of February, 1999, in Tallahassee, Leon County, Florida. CHARLES C. ADAMS Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 SUNCOM 278-9675 Fax Filing (850) 921-6847 www.doah.state.fl.us Filed with the Clerk of the Division of Administrative Hearings this 22nd day of February, 1999.
The Issue The issues to be decided are (i) whether Emergency Rule 64ER17-7(1)(b)-(d) constitutes an invalid exercise of delegated legislative authority, and (ii) whether Respondent's scoring methodology, which comprises several policies and procedures for determining the aggregate scores of the nurseries that applied for Dispensing Organization licenses in 2015, constitutes an unadopted rule.
Findings Of Fact BACKGROUND AND PARTIES Respondent Florida Department of Health (the "Department" or "DOH") is the agency responsible for administering and enforcing laws that relate to the general health of the people of the state. The Department's jurisdiction includes the state's medical marijuana program, which the Department oversees. Art. X, § 29, Fla. Const.; § 381.986, Fla. Stat. Enacted in 2014, section 381.986, Florida Statutes (2015) (the "Noneuphoric Cannabis Law"), legalized the use of low-THC cannabis by qualified patients having specified illnesses, such as cancer and debilitating conditions that produce severe and persistent seizures and muscle spasms. The Noneuphoric Cannabis Law directed the Department to select one dispensing organization ("DO") for each of five geographic areas referred to as the northwest, northeast, central, southwest, and southeast regions of Florida. Once licensed, a regional DO would be authorized to cultivate, process, and sell medical marijuana, statewide, to qualified patients. Section 381.986(5)(b), Florida Statutes (2015), prescribed various conditions that an applicant would need to meet to be licensed as a DO, and it required the Department to "develop an application form and impose an initial application and biennial renewal fee." DOH was, further, granted authority to "adopt rules necessary to implement" the Noneuphoric Cannabis Law. § 381.986(5)(d), Fla. Stat. (2015). Accordingly, the Department's Office of Compassionate Use ("OCU"), which is now known as the Office of Medical Marijuana Use, adopted rules under which a nursery could apply for a DO license. Incorporated by reference in these rules is a form of an Application for Low-THC Cannabis Dispensing Organization Approval ("Application"). See Fla. Admin. Code R. 64-4.002 (incorporating Form DH9008-OCU-2/2015). To apply for one of the initial DO licenses, a nursery needed to submit a completed Application, including the $60,063.00 application fee, no later than July 8, 2015.1/ See Fla. Admin. Code R. 64-4.002(5). Petitioner Nature's Way of Miami, Inc. ("Nature's Way"), is a nursery located in Miami, Florida, which grows and sells tropical plants to big box retailers throughout the nation. Nature's Way timely applied to the Department in 2015 for licensure as a DO in the southeast region. THE 2015 DO APPLICATION CYCLE These rule challenges arise from the Department's intended denial of Nature's Way's October 19, 2017, application for registration as a medical marijuana treatment center ("MMTC"), which is the name by which DOs are now known. Nature's Way asserts that it qualifies for licensure as an MMTC because it meets the newly created "One Point Condition," which can be satisfied only by a nursery, such as Nature's Way, whose 2015 application for licensure as a DO was evaluated, scored, and not approved as of the enactment, in 2017, of legislation that substantially overhauled the Noneuphoric Cannabis Law. See Ch. 2017-232, Laws of Fla. The current iteration of section 381.986, in effect as of this writing, will be called the "Medical Marijuana Law." The One Point Condition operates retroactively in that it establishes a previously nonexistent basis for licensure that depends upon pre-enactment events. This is analogous to the legislative creation of a new cause of action, involving as it does the imposition of a new duty (to issue licenses) on the Department and the bestowal of a new right (to become licensed) on former applicants based on their past actions. Facts surrounding the inaugural competition under the Noneuphoric Cannabis Law for regional DO licenses are material, therefore, to the determination not only of whether an applicant for licensure as an MMTC under the Medical Marijuana Law meets the One Point Condition, but also of the (in)validity of the emergency rule at issue, and the (il)legality of the agency statements alleged to be rules by definition, upon which the Department relies in applying the One Point Condition. To understand the issues at hand, it is essential first to become familiar with the evaluation and scoring of, and the agency actions with respect to, the applications submitted during the 2015 DO application cycle. The Competitive, Comparative Evaluation As stated in the Application, OCU viewed its duty to select five regional DOs as requiring OCU to choose "the most dependable, most qualified" applicant in each region "that can consistently deliver high-quality" medical marijuana. For ease of reference, such an applicant will be referred to as the "Best" applicant for short. Conversely, an applicant not chosen by OCU as "the most dependable, most qualified" applicant in a given region will be called, simply, "Not Best." Given the limited number of available DO licenses under the Noneuphoric Cannabis Law, the 2015 application process necessarily entailed a competition. As the Application explained, applicants were not required to meet any "mandatory minimum criteria set by the OCU," but would be evaluated comparatively in relation to the "other Applicants" for the same regional license, using criteria "drawn directly from the Statute." Clearly, the comparative evaluation would require the item-by-item comparison of competing applicants, where the "items" being compared would be identifiable factors drawn from the statute and established in advance. Contrary to the Department's current litigating position, however, it is not an intrinsic characteristic of a comparative evaluation that observations made in the course thereof must be recorded using only comparative or superlative adjectives (e.g., least qualified, qualified, more qualified, most qualified).2/ Moreover, nothing in the Noneuphoric Cannabis Law, the Application, or Florida Administrative Code Rule 64-4.002 stated expressly, or necessarily implied, that in conducting the comparative evaluation, OCU would not quantify (express numerically an amount denoting) the perceived margins of difference between competing applications. Quite the opposite is true, in fact, because, as will be seen, rule 64-4.002 necessarily implied, if it did not explicitly require, that the applicants would receive scores which expressed their relative merit in interpretable intervals. Specifically, the Department was required to "substantively review, evaluate, and score" all timely submitted and complete applications. Fla. Admin. Code R. 64-4.002(5)(a). This evaluation was to be conducted by a three-person committee (the "Reviewers"), each member of which had the duty to independently review and score each application. See Fla. Admin. Code R. 64-4.002(5)(b). The applicant with the "highest aggregate score" in each region would be selected as the Department's intended licensee for that region. A "score" is commonly understood to be "a number that expresses accomplishment (as in a game or test) or excellence (as in quality) either absolutely in points gained or by comparison to a standard." See "Score," Merriam-Webster.com, http://www.merriam-webster.com (last visited May 30, 2018). Scores are expressed in cardinal numbers, which show quantity, e.g., how many or how much. When used as a verb in this context, the word "score" plainly means "to determine the merit of," or to "grade," id., so that the assigned score should be a cardinal number that tells how much quality the graded application has as compared to the competing applications. The language of the rule leaves little or no doubt that the Reviewers were supposed to score the applicants in a way that quantified the differences between them, rather than with superlatives such as "more qualified" and "most qualified" (or numbers that merely represented superlative adjectives). By rule, the Department had identified the specific items that the Reviewers would consider during the evaluation. These items were organized around five subjects, which the undersigned will refer to as Topics. The five Topics were Cultivation, Processing, Dispensing, Medical Director, and Financials. Under the Topics of Cultivation, Processing, and Dispensing were four Subtopics (the undersigned's term): Technical Ability; Infrastructure; Premises, Resources, Personnel; and Accountability. In the event, the 12 Topic-Subtopic combinations (e.g., Cultivation-Technical Ability, Cultivation- Infrastructure), together with the two undivided Topics (i.e., Medical Director and Financials), operated as 14 separate evaluation categories. The undersigned refers to these 14 categories as Domains. The Department assigned a weight (by rule) to each Topic, denoting the relative importance of each in assessing an applicant's overall merit. The Subtopics, in turn, were worth 25% of their respective Topics' scores, so that a Topic's raw or unadjusted score would be the average of its four Subtopics' scores, if it had them. The 14 Domains and their associated weights are shown in the following table: CULTIVATION 30% 1. Cultivation – Technical Ability 25% out of 30% 2. Cultivation – Infrastructure 25% out of 30% 3. Cultivation – Premises, Resources, Personnel 25% out of 30% 4. Cultivation – Accountability 25% out of 30% PROCESSING 30% 5. Processing – Technical Ability 25% out of 30% 6. Processing – Infrastructure 25% out of 30% 7. Processing: Premises, Resources, Personnel 25% out of 30% 8. Processing: Accountability 25% out of 30% DISPENSING 15% 9. Dispensing: Technical Ability 25% out of 15% 10. Dispensing: Infrastructure 25% out of 15% 11. Dispensing: Premises, Resources, Personnel 25% out of 15% 12. Dispensing: Accountability 25% out of 15% 13. MEDICAL DIRECTOR 5% 14. FINANCIALS 20% If there were any ambiguity in the meaning of the word "score" as used in rule 64-4.002(5)(b), the fact of the weighting scheme removes all uncertainty, because in order to take a meaningful percentage (or fraction) of a number, the number must signify a divisible quantity, or else the reduction of the number, x, to say, 20% of x, will not be interpretable. Some additional explanation here might be helpful. If the number 5 is used to express how much of something we have, e.g., 5 pounds of flour, we can comprehend the meaning of 20% of that value (1 pound of flour). On the other hand, if we have coded the rank of "first place" with the number 5 (rather than, e.g., the letter A, which would be equally functional as a symbol), the meaning of 20% of that value is incomprehensible (no different, in fact, than the meaning of 20% of A). To be sure, we could multiply the number 5 by 0.20 and get 1, but the product of this operation, despite being mathematically correct (i.e., true in the abstract, as a computational result), would have no contextual meaning. This is because 20% of first place makes no sense. Coding the rank of first place with the misleading symbol of "5 points" would not help, either, because the underlying referent——still a position, not a quantity——is indivisible no matter what symbol it is given.3/ We can take this analysis further. The weighting scheme clearly required that the points awarded to an applicant for each Topic must contribute a prescribed proportionate share both to the applicant's final score per Reviewer, as well as to its aggregate score. For example, an applicant's score for Financials had to be 20% of its final Reviewer scores and 20% of its aggregate score, fixing the ratio of unweighted Financials points to final points (both Reviewer and aggregate) at 5:1. For this to work, a point scale having fixed boundaries had to be used, and the maximum number of points available for the final scores needed to be equal to the maximum number of points available for the raw (unweighted) scores at the Topic level. In other words, to preserve proportionality, if the applicants were scored on a 100-point scale, the maximum final score had to be 100, and the maximum raw score for each of the five Topics needed to be 100, too. The reasons for this are as follows. If there were no limit to the number of points an applicant could earn at the Topic level (like a baseball game), the proportionality of the weighting scheme could not be maintained; an applicant might run up huge scores in lower-weighted Topics, for example, making them proportionately more important to its final score than higher-weighted Topics. Similarly, if the maximum number of points available at the Topic level differed from the maximum number of points available as a final score, the proportionality of the weighting scheme (the prescribed ratios) would be upset, obviously, because, needless to say, 30% of, e.g., 75 points is not equal to 30% of 100 points. If a point scale is required to preserve proportionality, and it is, then so, too, must the intervals between points be the same, for all scores, in all categories, or else the proportionality of the weighting scheme will fail. For a scale to be uniform and meaningful, which is necessary to maintain the required proportionality, the points in it must be equidistant from each other; that is, the interval between 4 and 5, for example, needs to be the same as the interval between 2 and 3, and the distance between 85 and 95 (if the scale goes that high) has to equal that between 25 and 35.4/ When the distances between values are known, the numbers are said to express interval data.5/ Unless the distances between points are certain and identical, the prescribed proportions of the weighting scheme established in rule 64-4.002 will be without meaning. Simply stated, there can be no sense of proportion without interpretable intervals. We cannot say that a 5:1 relationship exists between two point totals (scores) if we have no idea what the distance is between 5 points and 1 point. The weighting system thus necessarily implied that the "scores" assigned by the Reviewers during the comparative evaluation would be numerical values (points) that (i) expressed quantity; (ii) bore some rational relationship to the amount of quality the Reviewer perceived in an applicant in relation to the other applicants; and (iii) constituted interval data. In other words, the rule unambiguously required that relative quality be counted (quantified), not merely coded. The Scoring Methodology: Interval Coding In performing the comparative evaluation of the initial applications filed in 2015, the Reviewers were required to use Form DH8007-OCU-2/2015, "Scorecard for Low-THC Cannabis Dispensing Organization Selection" (the "Scorecard"), which is incorporated by reference in rule 64-4.002(5)(a). There are no instructions on the Scorecard. The Department's rules are silent to how the Reviewers were supposed to score applications using the Scorecard, and they provide no process for generating aggregate scores from Reviewer scores. To fill these gaps, the Department devised several policies that governed its free-form decision-making in the run- up to taking preliminary agency action on the applications. Regarding raw scores, the Department decided that the Reviewers would sort the applications by region and then rank the applications, from best to worst, on a per-Domain basis, so that each Reviewer would rank each applicant 14 times (the "Ranking Policy"). An applicant's raw Domanial score would be its position in the ranking, from 1 to x, where x was both (i) equal to the number of applicants within the region under review and (ii) the number assigned to the rank of first place (or Best). In other words, the Reviewer's judgments as to the descending order of suitability of the competing applicants, per Domain, were symbolized or coded with numbers that the Department called "rank scores," and which were thereafter used as the applicants' raw Domanial scores. To be more specific, in a five-applicant field such as the southeast region, the evaluative judgments of the Reviewers were coded as follows: Evaluative Judgment Symbol ("Rank Score") Best qualified applicant ("Best") 5 points Less qualified than the best qualified applicant, but better qualified than all other applicants ("Second Best") 4 points Less qualified than two better qualified applicants, but better qualified than all other applicants ("Third Best") 3 points Less qualified than three better qualified applicants, but better qualified than all other applicants ("Fourth Best") 2 points Less qualified than four better qualified applicants ("Fifth Best") 1 point The Department's unfortunate decision to code the Reviewers' qualitative judgments regarding positions in rank orders with symbols that look like quantitative judgments regarding amounts of quality led inexorably to extremely misleading results. The so-called "rank scores" give the false impression of interval data, tricking the consumer (and evidently the Department, too) into believing that the distance between scores is certain and the same; that, in other words, an applicant with a "rank score" of 4 is 2 points better than an applicant with a "rank score" of 2. If this deception had been intentional (and, to be clear, there is no evidence it was), we could fairly call it fraud. Even without bad intent, the decision to code positions in ranked series with "scores" expressed as "points" was a colossal blunder that turned the scoring process into a dumpster fire. Before proceeding, it must be made clear that an applicant's being ranked Best in a Domain meant only that, as the highest-ranked applicant, it was deemed more suitable, by some unknown margin, than all the others within the group. By the same token, to be named Second Best meant only that this applicant was less good, in some unknown degree, than the Best applicant, and better, in some unknown degree, than the Third Best and remaining, lower-ranked applicants. The degree of difference in suitability between any two applicants in any Domanial ranking might have been a tiny sliver or a wide gap, even if they occupied adjacent positions, e.g., Second Best and Third Best. The Reviewers made no findings with respect to degrees of difference. Moreover, it cannot truthfully be claimed that the interval between, say, Second Best and Third Best is the same as that between Third Best and Fourth Best, for there exists no basis in fact for such a claim. In sum, the Department's Domanial "rank scores" merely symbolized the applicants' positions in sets of ordered applications. Numbers which designate the respective places (ranks) occupied by items in an ordered list are called ordinal numbers. The type of non-metric data that the "rank scores" symbolize is known as ordinal data, meaning that although the information can be arranged in a meaningful order, there is no unit or meter by which the intervals between places in the ranking can be measured. Because it is grossly misleading to refer to positions in a ranking as "scores" counted in "points," the so-called "rank scores" will hereafter be referred to as "Ordinals"——a constant reminder that we are working with ordinal data. This is important to keep in mind because, as will be seen, there are limits on the kinds of mathematical manipulation that can appropriately be carried out with ordinal data. The Department's policy of coding positions in a rank order with "rank scores" expressed as "points" will be called the "Interval Coding Policy." In conducting the evaluation, the Reviewers followed the Ranking Policy and Interval Coding Policy (collectively, the "Rank Scores Policies"). The Computational Methodology: Interval Statements and More Once the Reviewers finished evaluating and coding the applications, the evaluative phase of the Department's free-form process was concluded. The Reviewers had produced a dataset of Domanial Ordinals——42 Domanial Ordinals for each applicant to be exact——that collectively comprised a compilation of information, stored in the scorecards. This universe of Domanial Ordinals will be called herein the "Evaluation Data." The Department would use the Evaluation Data in the next phase of its free-form process as grounds for computing the applicants' aggregate scores. Rule 64-4.002(5)(b) provides that "scorecards from each reviewer will be combined to generate an aggregate score for each application. The Applicant with the highest aggregate score in each dispensing region shall be selected as the region's Dispensing Organization." Notice that the rule here switches to the passive voice. The tasks of (i) "combin[ing]" scorecards to "generate" aggregate scores and of (ii) "select[ing]" regional DOs were not assigned to the Reviewers, whose work was done upon submission of the scorecards. As mentioned previously, the rule does not specify how the Evaluation Data will be used to generate aggregate scores. The Department formulated extralegal policies6/ for this purpose, which can be stated as follows: (i) the Ordinals, which in actuality are numeric code for uncountable information content, shall be deemed real (counted) points, i.e., equidistant units of measurement on a 5-point interval scale (the "Deemed Points Policy"); (ii) in determining aggregate scores, the three Reviewer scores will be averaged instead of added together, so that "aggregate score" means "average Reviewer score" (the "Aggregate Definition"); and (iii) the results of mathematical computations used to determine weighted scores at the Reviewer level and, ultimately, the aggregate scores themselves will be carried out to the fourth decimal place (the "Four Decimal Policy"). Collectively, these three policies will be referred to as the "Generation Policies." The Department's "Scoring Methodology" comprises the Rank Scores Policies and the Generation Policies. The Department's computational process for generating aggregate scores operated like this. For each applicant, a Reviewer final score was derived from each Reviewer, using that Reviewer's 14 Domanial Ordinals for the applicant. For each of the subdivided Topics (Cultivation, Processing, and Dispensing), the mean of the Reviewer's four Domanial Ordinals for the applicant (one Domanial Ordinal for each Subtopic) was determined by adding the four numbers (which, remember, were whole numbers as discussed above) and dividing the sum by 4. The results of these mathematical operations were reported to the second decimal place. (The Reviewer raw score for each of the subdivided Topics was, in other words, the Reviewer's average Subtopic Domanial Ordinal.) For the undivided Topics of Medical Director and Financials, the Reviewer raw score was simply the Domanial Ordinal, as there was only one Domanial Ordinal per undivided Topic. The five Reviewer raw Topic scores (per Reviewer) were then adjusted to account for the applicable weight factor. So, the Reviewer raw scores for Cultivation and Processing were each multiplied by 0.30; raw scores for Dispensing were multiplied by 0.15; raw scores (Domanial Ordinals) for Medical Director were multiplied by 0.05; and raw scores (Domanial Ordinals) for Financials were multiplied by 0.20. These operations produced five Reviewer weighted-Topic scores (per Reviewer), carried out (eventually) to the fourth decimal place. The Reviewer final score was computed by adding the five Reviewer weighted-Topic scores. Thus, each applicant wound up with three Reviewer final scores, each reported to the fourth decimal place pursuant to the Four Decimal Policy. The computations by which the Department determined the three Reviewer final scores are reflected (but not shown) in a "Master Spreadsheet"7/ that the Department prepared. Comprising three pages (one for each Reviewer), the Master Spreadsheet shows all of the Evaluation Data, plus the 15 Reviewer raw Topic scores per applicant, and the three Reviewer final scores for each applicant. Therein, the Reviewer final scores of Reviewer 2 and Reviewer 3 were not reported as numbers having five significant digits, but were rounded to the nearest hundredth. To generate an applicant's aggregate score, the Department, following the Aggregate Definition, computed the average Reviewer final score by adding the three Reviewer final scores and dividing the sum by 3. The result, under the Four Decimal Policy, was carried out the ten-thousandth decimal point. The Department referred to the aggregate score as the "final rank" in its internal worksheets. The Department further assigned a "regional rank" to each applicant, which ordered the applicants, from best to worst, based on their aggregate scores. Put another way, the regional rank was an applicant's Ultimate Ordinal. The Reviewer final scores and the "final ranks" (all carried out to the fourth decimal place), together with the "regional ranks," are set forth in a table the Department has labeled its November 2015 Aggregated Score Card (the "Score Card"). The Score Card does not contain the Evaluation Data. Preliminary Agency Actions Once the aggregate scores had been computed, the Department was ready to take preliminary agency action on the applications. As to each application, the Department made a binary decision: Best or Not Best. The intended action on the applications of the five Best applicants (one per region), which were identified by their aggregate scores (highest per region), would be to grant them. Each of the Not Best applicants, so deemed due to their not having been among the highest scored applicants, would be notified that the Department intended to deny its application. The ultimate factual determination that the Department made for each application was whether the applicant was, or was not, the most dependable, most qualified nursery as compared to the alternatives available in a particular region. Clear Points of Entry Letters dated November 23, 2015, were sent to the applicants informing them either that "your application received the highest score" and thus is granted, or that because "[you were] not the highest scored applicant in [your] region, your application . . . is denied," whichever was the case. The letters contained a clear point of entry, which concluded with the usual warning that the "[f]ailure to file a petition within 21 days shall constitute a waiver of the right to a hearing on this agency action." 8/ (Emphasis added). Nature's Way decided not to request a hearing in 2015, and therefore it is undisputed that the Department's proposed action, i.e., the denial of Nature's Way's application because the applicant was not deemed to be the most dependable, most qualified nursery for purposes of selecting a DO for the southeast region, became final agency action without a formal hearing, the right to which Nature's Way elected to waive. The Department argues that Nature's Way thereby waived, forever and for all purposes, the right to a hearing on the question of whether its aggregate score of 2.8833 and Costa's aggregate score of 4.4000 (highest in the southeast region)——which the Department generated using the Scoring Methodology——are, in fact, true as interval statements of quantity. (Note that if these scores are false as interval data, as Nature's Way contends, then the statement that Costa's score exceeds Nature's Way's score by 1.5167 points is false, also, because it is impossible to calculate a true, interpretable difference (interval) between two values unless those values are expressions of quantified data. Simply put, you cannot subtract Fourth Best from Best.) The Department's waiver argument, properly understood, asserts that Nature's Way is barred by administrative finality from "relitigating" matters, such as the truth of the aggregate scores as quantifiable facts, which were supposedly decided conclusively in the final agency action on its DO application in 2015. To successfully check Nature's Way with the affirmative defense of administrative finality, the Department needed to prove that the truth of the aggregate scores, as measurable quantities, was actually adjudicated (or at least judicable) in 2015, so that the numbers 2.8833 and 4.4000 are now incontestably true interval data, such that one figure can meaningfully be subtracted from the other for purposes of applying the One Point Condition. The Department's affirmative defense of collateral estoppel/issue preclusion was rejected in the related disputed- fact proceeding, which is the companion to this litigation, based on the undersigned's determination that the truth of the aggregate scores as statements of fact expressing interval data had never been previously adjudicated as between the Department and Nature's Way. See Nature's Way Nursery of Miami, Inc. v. Dep't of Health, Case No. 18-0721 (Fla. DOAH June 15, 2018). The Ambiguity of the Aggregate Scores There is a strong tendency to look at a number such as 2.8833 and assume that it is unambiguous——and, indeed, the Department is unquestionably attempting to capitalize on that tendency. But numbers can be ambiguous.9/ The aggregate scores are, clearly, open to interpretation. To begin, however, it must be stated up front that there is no dispute about the existence of the aggregate scores. It is an undisputed historical fact, for example, that Nature's Way had a final ranking (aggregate score) of 2.8833 as computed by the Department in November 2015. There is likewise no dispute that Costa's Department-computed aggregate score was 4.4000. In this sense, the scores are historical facts—— relevant ones, too, since an applicant needed to have had an aggregate score in 2015 to take advantage of the One Point Condition enacted in 2017. The existence of the scores, however, is a separate property from their meaning. Clearly, the aggregate scores that exist from history purport to convey information about the applicants; in effect, they are statements. The ambiguity arises from the fact that each score could be interpreted as having either of two different meanings. On the one hand, an aggregate score could be understood as a numerically coded non- quantity, namely a rank. In other words, the aggregate scores could be interpreted reasonably as ordinal data. On the other hand, an aggregate score could be understood as a quantified measurement taken in units of equal value, i.e., interval data. In 2015, the Department insisted (when it suited its purposes) that the aggregate scores were numeric shorthand for its discretionary value judgments about which applicants were best suited, by region, to be DOs, reflecting where the applicants, by region, stood in relation to the best suited applicants and to each other. The Department took this position because it wanted to limit the scope of the formal hearings requested by disappointed applicants to reviewing its decisions for abuse of discretion. Yet, even then, the Department wanted the aggregate scores to be seen as something more rigorously determined than a discretionary ranking. Scores such as 2.8833 and 3.2125 plainly connote a much greater degree of precision than "these applicants are less qualified than others." Indeed, in one formal hearing, the Department strongly implied that the aggregate scores expressed interval data, arguing that they showed "the [Department's position regarding the] order of magnitude" of the differences in "qualitative value" between the applicants, so that a Fourth Best applicant having a score of 2.6458 was asserted to be "far behind" the highest-scored applicant whose final ranking was 4.1042.10/ A ranking, of course, expresses order but not magnitude; interval data, in contrast, expresses both order and magnitude, and it is factual in nature, capable of being true or false. In short, as far as the meaning of the aggregate scores is concerned, the Department has wanted to have it both ways. Currently, the Department is all-in on the notion that the aggregate scores constitute precise interval data, i.e., quantified facts. In its Proposed Recommended Order in Case No. 18-0721,11/ on page 11, the Department argues that "Nature's Way does not meet the within-one-point requirement" because "Nature's Way's Final Rank [aggregate score of 2.8833] is 1.5167 points less than the highest Final Rank [Cost's aggregate score, 4.4000] in its region." This is a straight-up statement of fact, not a value judgment or policy preference. Moreover, it is a statement of fact which is true only if the two aggregate scores being compared (2.8833 and 4.4000), themselves, are true statements of quantifiable fact about the respective applicants. The Department now even goes so far as to claim that the aggregate score is the precise and true number (quantity) of points that an applicant earned as a matter of fact. On page 6 of its Proposed Final Order, the Department states that Costa "earned a Final Rank of 4.4000" and that Nature's Way had an "earned Final Rank of 2.8833." In this view, the scores tell us not that, in the Department's discretionary assignment of value, Costa was better suited to be the DO for the southeast region, but rather that (in a contest, it is insinuated, the Department merely refereed) Costa outscored Nature's Way by exactly 1.5167 points——and that the points have meaning as equidistant units of measurement. The Department is plainly using the aggregate scores, today, as interval statements of quantifiable fact, claiming that Nature's Way "earned" exactly 2.8833 points on a 5-point scale where each point represents a standard unit of measurement, while Costa "earned" 4.4000 points; this, again, is the only way it would be correct to say that Costa was 1.5167 points better than Nature's Way. Indeed, Emergency Rule 64ER17-7 (the "Emergency Rule") purports to codify this interpretation of the aggregate scores——and to declare that the 2015 aggregate scores are true as interval data. ENACTMENT OF THE MEDICAL MARIJUANA LAW Effective January 3, 2017, Article X of the Florida Constitution was amended to include a new section 29, which addresses medical marijuana production, possession, dispensing, and use. Generally speaking, section 29 expands access to medical marijuana beyond the framework created by the Florida Legislature in 2014. To implement the newly adopted constitutional provisions and "create a unified regulatory structure," the legislature enacted the Medical Marijuana Law, which substantially revised section 381.986 during the 2017 Special Session. Ch. 2017-232, § 1, Laws of Fla. Among other things, the Medical Marijuana Law establishes a licensing protocol for ten new MMTCs. The relevant language of the new statute states: (8) MEDICAL MARIJUANA TREATMENT CENTERS.— (a) The department shall license medical marijuana treatment centers to ensure reasonable statewide accessibility and availability as necessary for qualified patients registered in the medical marijuana use registry and who are issued a physician certification under this section. * * * The department shall license as medical marijuana treatment centers 10 applicants that meet the requirements of this section, under the following parameters: As soon as practicable, but no later than August 1, 2017, the department shall license any applicant whose application was reviewed, evaluated, and scored by the department and which was denied a dispensing organization license by the department under former s. 381.986, Florida Statutes 2014; which had one or more administrative or judicial challenges pending as of January 1, 2017, or had a final ranking within one point of the highest final ranking in its region under former s. 381.986, Florida Statutes 2014; which meets the requirements of this section; and which provides documentation to the department that it has the existing infrastructure and technical and technological ability to begin cultivating marijuana within 30 days after registration as a medical marijuana treatment center. § 381.986, Fla. Stat. (Emphasis added: The underscored provision is the One Point Condition). The legislature granted the Department rulemaking authority, as needed, to implement the provisions of section 381.986(8). § 381.986(8)(k), Fla. Stat. In addition, the legislature authorized the Department to adopt emergency rules pursuant to section 120.54(4), as necessary to implement section 381.986, without having to find an actual emergency, as otherwise required by section 120.54(4)(a). Ch. 2017-232, § 14, at 45, Laws of Fla. IMPLEMENTATION OF THE ONE POINT CONDITION AND ADOPTION OF THE EMERGENCY RULE The One Point Condition went into effect on June 23, 2017. Ch. 2017-232, § 20, Laws of Fla. Thereafter, the Department issued a license to Sun Bulb Nursery (a 2015 DO applicant in the southwest region), because the Department concluded that Sun Bulb's final ranking was within one point of the highest final ranking in the southwest region.12/ Keith St. Germain Nursery Farms ("KSG"), like Nature's Way a 2015 DO applicant for the southeast region, requested MMTC registration pursuant to the One Point Condition in June 2017. In its request for registration, KSG asserted that the One Point Condition is ambiguous and proposed that the Department either calculate the one-point difference based on the regional ranks set forth in the Score Card (KSG was the regional Second Best, coded as Ultimate Ordinal 4) or round off the spurious decimal points in the aggregate scores when determining the one-point difference. The Department preliminarily denied KSG's request for MMTC registration in August 2017. In its notice of intent, the Department stated in part: The highest-scoring entity in the Southeast Region, Costa Nursery Farms, LLC, received a final aggregate score of 4.4000. KSG received a final aggregate score of 3.2125. Therefore, KSG was not within one point of Costa Farms. KSG requested a disputed-fact hearing on this proposed agency action and also filed with DOAH a Petition for Formal Administrative Hearing and Administrative Determination Concerning Unadopted Rules, initiating Keith St. Germain Nursery Farms v. Florida Department of Health, DOAH Case No. 17-5011RU ("KSG's Section 120.56(4) Proceeding"). KSG's Section 120.56(4) Proceeding, which Nature's Way joined as a party by intervention, challenged the legality of the Department's alleged unadopted rules for determining which of the 2015 DO applicants were qualified for licensure pursuant to the One Point Condition. Faced with the KSG litigation, the Department adopted Emergency Rule 64ER17-3, which stated in relevant part: For the purposes of implementing s. 381.986(8)(a)2.a., F.S., the following words and phrases shall have the meanings indicated: Application – an application to be a dispensing organization under former s. 381.986, F.S. (2014), that was timely submitted in accordance with Rule 64- 4.002(5) of the Florida Administrative Code (2015). Final Ranking – an applicant's aggregate score for a given region as provided in the column titled "Final Rank" within the November 2015 Aggregated Score Card, incorporated by reference and available at [hyperlink omitted], as the final rank existed on November 23, 2015. Highest Final Ranking – the final rank with the highest point value for a given region, consisting of an applicant's aggregate score as provided in the column titled "Final Rank" within the November 2015 Aggregated Score Card, as the final rank existed on November 23, 2015. Within One Point – one integer (i.e., whole, non-rounded number) carried out to four decimal points (i.e., 1.0000) by subtracting an applicant's final ranking from the highest final ranking in the region for which the applicant applied. Qualified 2015 Applicant – an individual or entity whose application was reviewed, evaluated, and scored by the department and that was denied a dispensing organization license under former s. 381.986, F.S. (2014) and either: (1) had one or more administrative or judicial challenges pending as of January 1, 2017; or had a final ranking within one point of the highest final ranking in the region for which it applied, in accordance with Rule 64-4.002(5) of the Florida Administrative Code (2015). The Department admits that not much analysis or thought was given to the development of this rule, which reflected the Department's knee-jerk conclusion that the One Point Condition's use of the term "final ranking" clearly and unambiguously incorporated the applicants' "aggregate scores" (i.e., "final rank" positions), as stated in the Score Card, into the statute. In any event, the rule's transparent purpose was to adjudicate the pending licensing dispute with KSG and shore up the Department's ongoing refusal (in Department of Health Case No. 2017-0232) to grant KSG a formal disputed-fact hearing on the proposed denial of its application. Naturally, the Department took the position that rule 64ER17-3 had settled all possible disputes of material fact, once and for all, as a matter of law. In a surprising about-face, however, on October 26, 2017, the Department entered into a settlement agreement with KSG pursuant to which the Department agreed to register KSG as an MMTC. The Department issued a Final Order Adopting Settlement Agreement with KSG on October 30, 2017. That same day (and in order to effectuate the settlement with KSG), the Department issued the Emergency Rule. The Emergency Rule amends former rule 64ER17-3 to expand the pool of Qualified 2015 Applicants by exactly one, adding KSG——not by name, of course, but by deeming all the regional Second Best applicants to be Within One Point. Because KSG was the only 2015 applicant ranked Second Best in its region that did not have an aggregate score within one point of its region's Best applicant in accordance with rule 64ER17-3, KSG was the only nursery that could take advantage of the newly adopted provisions. As relevant, the Emergency Rule provides as follows: This emergency rule supersedes the emergency rule 64ER17-3 which was filed and effective on September 28, 2017. For the purposes of implementing s. 381.986(8)(a)2.a., F.S., the following words and phrases shall have the meanings indicated: Application – an application to be a dispensing organization under former s. 381.986, F.S. (2014), that was timely submitted in accordance with Rule 64- 4.002(5) of the Florida Administrative Code (2015). Final Ranking – an applicant's aggregate score for a given region as provided in the column titled "Final Rank" or the applicant's regional rank as provided in the column titled "Regional Rank" within the November 2015 Aggregated Score Card, incorporated by reference and available at [hyperlink omitted], as the final rank existed on November 23, 2015. Highest Final Ranking – the final rank with the highest point value for a given region, consisting of an applicant's aggregate score as provided in the column titled "Final Rank" or the applicant's regional rank as provided in the column titled "Regional Rank" within the November 2015 Aggregated Score Card, as the final rank existed on November 23, 2015. Within One Point – for the aggregate score under the column "Final Rank" one integer (i.e., whole, non-rounded number) carried out to four decimal points (i.e., 1.0000) or for the regional rank under the column "Regional Rank" one whole number difference, by subtracting an applicant's final ranking from the highest final ranking in the region for which the applicant applied. Qualified 2015 Applicant – an individual or entity whose application was reviewed, evaluated, and scored by the department and that was denied a dispensing organization license under former s. 381.986, F.S. (2014) and either: (1) had one or more administrative or judicial challenges pending as of January 1, 2017; or had a final ranking within one point of the highest final ranking in the region for which it applied, in accordance with Rule 64-4.002(5) of the Florida Administrative Code (2015). (Emphasis added). In a nutshell, the Emergency Rule provides that an applicant meets the One Point Condition if either (i) the difference between its aggregate score and the highest regional aggregate score, as those scores were determined by the Department effective November 23, 2015, is less than or equal to 1.0000; or (ii) its regional rank, as determined by the Department effective November 23, 2015, is Second Best. A number of applicants satisfy both criteria, e.g., 3 Boys, McCrory's, Chestnut Hill, and Alpha (northwest region). Some, in contrast, meet only one or the other. Sun Bulb, Treadwell, and Loop's, for example, meet (i) but not (ii). KSG, alone, meets (ii) but not (i). The Department has been unable to come up with a credible, legally cohesive explanation for the amendments that distinguish the Emergency Rule from its predecessor. On the one hand, Christian Bax testified that KSG had persuaded the Department that "within one point" meant, for purposes of the One Point Condition, Second Best (or "second place"), and that this reading represented a reasonable interpretation of a "poorly crafted sentence" using an "unartfully crafted term," i.e., "final ranking." On the other hand, the Department argues in its Proposed Final Order (on page 17) that the One Point Condition's "plain language reflects the legislature's intent that the 'second-best' applicant in each region (if otherwise qualified) be licensed as an MMTC." (Emphasis added). Logically, of course, the One Point Condition cannot be both "poorly crafted" (i.e., ambiguous) and written in "plain language" (i.e., unambiguous); legally, it must be one or the other. Put another way, the One Point Condition either must be construed, which entails a legal analysis known as statutory interpretation that is governed by well-known canons of construction and results in a legal ruling declaring the meaning of the ambiguous terms, or it must be applied according to its plain language, if (as a matter of law) it is found to be unambiguous. Obviously, as well, the One Point Condition, whether straightforward or ambiguous, cannot mean both within one point and within one place, since these are completely different statuses. If the statute is clear and unambiguous, only one of the alternatives can be correct; if ambiguous, either might be permissible, but not both simultaneously. By adopting the Emergency Rule, the Department took a position in direct conflict with the notion that the One Point Condition is clear and unambiguous; its reinterpretation of the statute is consistent only with the notion that the statute is ambiguous, and its present attempt to disown that necessarily implicit conclusion is rejected. The irony is that the Department surrendered the high ground of statutory unambiguity, which it initially occupied and stoutly defended, to take up an indefensible position, where, instead of choosing between two arguably permissible, but mutually exclusive, interpretations, as required, it would adopt both interpretations. The only reasonable inference the undersigned can draw from the Department's bizarre maneuver is that the Emergency Rule is not the product of high-minded policy making but rather a litigation tactic, which the Department employed as a necessary step to resolve the multiple disputes then pending between it and KSG. The Emergency Rule was adopted to adjudicate the KSG disputes in KSG's favor, supplanting the original rule that was adopted to adjudicate the same disputes in the Department's favor. THE IRRATIONALITY OF THE SCORING METHODOLOGY The Department committed a gross conceptual error when it decided to treat ordinal data as interval data under its Interval Coding and Deemed Points Policies. Sadly, there is no way to fix this problem retroactively; no formula exists for converting or translating non-metric data such as rankings (which, for the most part, cannot meaningfully be manipulated mathematically) into quantitative data. Further, the defect in the Department's "scoring" process has deprived us of essential information, namely, actual measurements. A Second Look at the Department's Scoring Methodology The Department's Scoring Methodology was described above. Nevertheless, for purposes of explicating just how arbitrary and capricious were the results of this process, and to shed more light on the issues of fact which the Department hopes the Emergency Rule has resolved before they can ever become grounds for a disputed-fact hearing, the undersigned proposes that the way the Department arrived at its aggregate scores be reexamined. It will be recalled that each applicant received 14 Ordinals from each reviewer, i.e., one Ordinal per Domain. These will be referred to as Domanial Ordinals. Thus, each applicant received, collectively, 12 Domanial Ordinals apiece for the Main Topics of Cultivation, Processing, and Dispensing; and three Domanial Ordinals apiece for the Main Topics of Medical Director and Financials, for a total of 42 Domanial Ordinals. These five sets of Domanial Ordinals will be referred to generally as Arrays, and specifically as the Cultivation Array, the Processing Array, the Dispensing Array, the MD Array, and the Financials Array. Domanial Ordinals that have been sorted by Array will be referred to, hereafter, as Topical Ordinals. So, for example, the Cultivation Array comprises 12 Topical Ordinals per applicant. A table showing the Arrays of the southeast region applicants is attached as Appendix A. Keeping our attention on the Cultivation Array, observe that if we divide the sum of the 12 Topical Ordinals therein by 12, we will have calculated the mean (or average) of these Topical Ordinals. This value will be referred to as the Mean Topical Ordinal or "MTO." For each applicant, we can find five MTOs, one apiece for the five Main Topics. So, each applicant has a Cultivation MTO, a Processing MTO, and so forth. As discussed, each Main Topic was assigned a weight, e.g., 30% for Cultivation, 20% for Financials. These five weights will be referred to generally as Topical Weights, and specifically as the Cultivation Topical Weight, the Processing Topical Weight, etc. If we reduce, say, the Cultivation MTO to its associated Cultivation Topical Weight (in other words, take 30% of the Cultivation MTO), we will have produced the weighted MTO for the Main Topic of Cultivation. For each applicant, we can find five weighted MTOs ("WMTO"), which will be called specifically the Cultivation WMTO, the Processing WMTO, etc. The sum of each applicant's five WMTOs equals what the Department calls the applicant's aggregate score or final rank. In other words, in the Department's scoring methodology, an MTO is functionally a "Topical raw score" and a WMTO is an "adjusted Topical score" or, more simply, a "Topical subtotal." Thus, we can say, alternatively, that the sum of an applicant's five Topical subtotals equals its DOH-assigned aggregate score. For those in a hurry, an applicant's WMTOs (or Topical subtotals) can be computed quickly by dividing the sum of the Topical Ordinals in each Array by the respective divisors shown in the following table: Dividend Divisor Quotient Sum of the Topical Ordinals in the CULTIVATION Array ÷ 40 - Cultivation WMTO Sum of the Topical Ordinals in the PROCESSING Array ÷ 40 - Processing WMTO Dividend Divisor Quotient Sum of the Topical Ordinals in the DISPENSING Array ÷ 80 - Dispensing WMTO Sum of the Ordinals in Topical the MD Array ÷ 60 - MD WMTO Sum of the Topical Ordinals in the FINANCIALS Array ÷ 15 - Financials WMTO To advance the discussion, it is necessary to introduce some additional concepts. We have become familiar with the Ordinal, i.e., a number that the Department assigned to code a particular rank (5, 4, 3, 2, or 1).13/ From now on, the symbol ? will be used to represent the value of an Ordinal as a variable. There is another value, which we can imagine as a concept, namely the actual measurement or observation, which, as a variable, we will call x. For our purposes, x is the value that a Reviewer would have reported if he or she had been asked to quantify (to the fourth decimal place) the amount of an applicant's suitability vis-à-vis the attribute in view on a scale of 1.0000 to 5.0000, with 5.0000 being "ideal" and 1.0000 meaning, roughly, "serviceable." This value, x, is a theoretical construct only because no Reviewer actually made any such measurements; such measurements, however, could have been made, had the Reviewers been required to do so. Indeed, some vague idea, at least, of x must have been in each Reviewer's mind every time he or she ranked the applicants, or else there would have been no grounds for the rankings. Simply put, a particular value x can be supposed to stand behind every Topical Ordinal because every Topical Ordinal is a function of x. Unfortunately, we do not know x for any Topical Ordinal. Next, there is the true value of x, for which we will give the symbol µ. This is a purely theoretical notion because it represents the value that would be obtained by a perfect measurement, and there is no perfect measurement of anything, certainly not of relative suitability to serve as an MMTC.14/ Finally, measurements are subject to uncertainty, which can be expressed in absolute or relative terms. The absolute uncertainty expresses the size of the range of values in which the true value is highly likely to lie. A measurement given as 150 ± 0.5 pounds tells us that the absolute uncertainty is 0.5 pounds, and that the true value is probably between 149.5 and 150.5 pounds (150 – 0.5 and 150 + 0.5). This uncertainty can be expressed as a percentage of the measured value, i.e., 150 pounds ± .33%, because 0.5 is .33% of 150. With that background out of the way, let's return to concept of the mean. The arithmetic mean is probably the most commonly used operation for determining the central tendency (i.e., the average or typical value) of a dataset. No doubt everyone reading this Order, on many occasions, has found the average of, say, four numbers by adding them together and dividing by 4. When dealing with interval data, the mean is interpretable because the interval is interpretable. Where the distance between 4 and 5, for example, is the same as that between 5 and 6, everyone understands that 4.5 is halfway between 4 and 5. As long as we know that 4.5 is exactly halfway between 4 and 5, the arithmetic mean of 4 and 5 (i.e., 4.5) is interpretable. The mean of a set of measurement results gives an estimate of the true value of the measurement, assuming there is no systematic error in the data. The greater the number of measurements, the better the estimate. Therefore, if, for example, we had in this case an Array of xs, then the mean of that dataset (x¯) would approximate µ, especially for the Cultivation, Processing, and Dispensing Arrays, which have 12 observations apiece. If the Department had used x¯ as the Topical raw score instead of the MTO, then its scoring methodology would have been free of systematic error. But the Department did not use x¯ as the Topical raw score. In the event, it had only Arrays of ?s to work with, so when the Department calculated the mean of an Array, it got the average of a set of Ordinals (?¯), not x¯. Using the mean as a measure of the central tendency of ordinal data is highly problematic, if not impermissible, because the information is not quantifiable. In this case, the Department coded the rankings with numbers, but the numbers (i.e., the Ordinals), not being units of measurement, were just shorthand for content that must be expressed verbally, not quantifiably. The Ordinals, that is, translate meaningfully only as words, not as numbers, as can be seen in the table at paragraph 27, supra. Because these numbers merely signify order, the distances between them have no meaning; the interval, it follows, is not interpretable. In such a situation, 4.5 does not signify a halfway point between 4 and 5. Put another way, the average of Best and Second Best is not "Second-Best-and-a- half," for the obvious reason that the notion is nonsensical. To give a real-life example, the three Topical Ordinals in Nature's Way's MD Array are 5, 3, and 2. The average of Best, Third Best, and Fourth Best is plainly not "Third-Best-and-a- third," any more than the average of Friday, Wednesday, and Tuesday is Wednesday-and-a-third. For these reasons, statisticians and scientists ordinarily use the median or the mode to measure the central tendency of ordinal data, generally regarding the mean of such data to be invalid or uninterpretable. The median is the middle number, which is determined by arranging the data points from lowest to highest, and identifying the one having the same number of data points on either side (if the dataset contains an odd number of data points) or taking the average of the two data points in the middle (if the dataset contains an even number of data points). The mode is the most frequently occurring number. (If no number repeats, then there is no mode, and if two or more numbers recur with the same frequency, then there are multiple modes.) We can easily compute the medians, modes, and means of the Topical Ordinals in each of the applicants' Arrays. They are set forth in the following table. Cultivation 30% Processing 30% Dispensing 15% Medical Director 5% Financials 20% Bill's Median Mode Mean 1 1 1.8333 Median Mode Mean 2 2 1.7500 Median Mode Mean 1 1 1.1667 Median Mode Mean 2 NA 2.0000 Median Mode Mean 1 1 1.0000 Costa Median Mode Mean 5 5 4.6667 Median Mode Mean 4.5 5 4.1667 Median Mode Mean 4 4 4.0000 Median Mode Mean 4 4 4.3333 Median Mode Mean 5 5 4.6667 Keith St. Germain Median Mode Mean 4 4 3.4167 Median Mode Mean 4 4 3.2500 Median Mode Mean 2 2 2.4167 Median Mode Mean 4 NA 3.6667 Median Mode Mean 3 3 3.3333 Nature's Way Median Mode Mean 3 4 3.0833 Median Mode Mean 3 3 2.5833 Median Mode Mean 3.5 3 3.6667 Median Mode Mean 3 NA 3.3333 Median Mode Mean 2 2 2.3333 Redland Median Mode Mean 2 2 2.2500 Median Modes Mean 3.5 3, 4, 5 3.4167 Median Mode Mean 5 5 4.1667 Median Mode Mean 2 NA 2.3333 Median Mode Mean 4 NA 3.6667 It so happens that the associated medians, modes, and means here are remarkably similar——and sometimes the same. The point that must be understood, however, is that the respective means, despite their appearance of exactitude when drawn out to four decimal places, tell us nothing more (if, indeed, they tell us anything) than the medians and the modes, namely whether an applicant was typically ranked Best, Second Best, etc. The median and mode of Costa's Cultivation Ordinals, for example, are both 5, the number which signifies "Best." This supports the conclusion that "Best" was Costa's average ranking under Cultivation. The mean of these same Ordinals, 4.6667, appears to say something more exact about Costa, but, in fact, it does not. At most, the mean of 4.6667 tells us only that Costa was typically rated "Best" in Cultivation. (Because there is no cognizable position of rank associated with the fraction 0.6667, the number 4.6667 must be rounded if it is to be interpreted.) To say that 4.6667 means that Costa outscored KSG by 1.2500 "points" in Cultivation, therefore, or that Costa was 37% more suitable than KSG, would be a serious and indefensible error, for these are, respectively, interval and ratio statements, which are never permissible to make when discussing ordinal data. As should by now be clear, ?¯ is a value having limited usefulness, if any, which cannot ever be understood, properly, as an estimate of µ. The Department, regrettably, treated ?¯ as if it were the same as x¯ and, thus, a reasonable approximation of µ, making the grievous conceptual mistakes of using ordinal data to make interval-driven decisions, e.g., whom to select for licensure when the "difference" between applicants was as infinitesimal as 0.0041 "points," as well as interval representations about the differences between applicants, such as, "Costa's aggregate score is 1.5167 points greater than Nature's Way's aggregate score." Due to this flagrant defect in the Department's analytical process, the aggregate scores which the Department generated are hopelessly infected with systematic error, even though the mathematical calculations behind the flawed scores are computationally correct. Dr. Cornew's Solution Any attempt to translate the Ordinals into a reasonable approximation of interval data is bound to involve a tremendous amount of inherent uncertainty. If we want to ascertain the x behind a particular ?, all we can say for sure is that: [(? – n) + 0.000n] = x = [(? + a) – 0.000a], where n represents the number of places in rank below ?, and a symbolizes the number of places in rank above ?. The Ordinals of 1 and 5 are partial exceptions, because 1 = x = 5. Thus, when ? = 5, we can say [(? – n) + 0.000n] = x = 5, and when ? = 1, we can say 1 = x = [(? + a) – 0.000a]. The table below should make this easier to see. Lowest Possible Value of x Ordinal ? Highest Possible Value of x 1.0004 5 5.0000 1.0003 4 4.9999 1.0002 3 4.9998 1.0001 2 4.9997 1.0000 1 4.9996 As will be immediately apparent, all this tells us is that x could be, effectively, any score from 1 to 5——which ultimately tells us nothing. Accordingly, to make any use of the Ordinals in determining an applicant's satisfaction of the One Point Condition, we must make some assumptions, to narrow the uncertainty. Nature's Way's expert witness, Dr. Ronald W. Cornew,15/ offers a solution that the undersigned finds to be credible. Dr. Cornew proposes (and the undersigned agrees) that, for purposes of extrapolating the scores (values of x) for a given applicant, we can assume that the Ordinals for every other applicant are true values (µ) of x, in other words, perfectly measured scores expressing interval data——a heroic assumption in the Department's favor. Under this assumption, if the subject applicant's Ordinal is the ranking of, say, 3, we shall assume that the adjacent Ordinals of the other applicants, 2 and 4, are true quantitative values. This, in turn, implies that the true value of the subject applicant's Ordinal, as a quantified score, is anywhere between 2 and 4, since all we know about the subject applicant is that the Reviewer considered it to be, in terms of relative suitability, somewhere between the applicants ranked Fourth Best (2) and Second Best (4). If we make the foregoing Department-friendly assumption that the other applicants' Ordinals are µ, then the following is true for the unseen x behind each of the subject applicant's ?s: [(? – 1) + 0.0001] = x = [(? + 1) – 0.0001]. The Ordinals of 1 and 5 are, again, partial exceptions. Thus, when ? = 5, we can say 4.0001 = x = 5, and when ? = 1, we can say 1 = x = 1.9999. Dr. Cornew sensibly rounds off the insignificant ten-thousandths of points, simplifying what would otherwise be tedious mathematical calculations, so that: Lowest Possible Value of x Ordinal ? Highest Possible Value of x 4 5 5 3 4 5 2 3 4 1 2 3 1 1 2 We have now substantially, albeit artificially, reduced the uncertainty involved in translating ?s to xs. Our assumption allows us to say that x = ? ± 1 except where only negative uncertainty exists (because x cannot exceed 5) and where only positive uncertainty exists (because x cannot be less than 1). It is important to keep in mind, however, that (even with the very generous, pro-Department assumption about other applicants' "scores") the best we can do is identify the range of values within which x likely falls, meaning that the highest values and lowest values are not alternatives; rather, the extrapolated score comprises those two values and all values in between, at once. In other words, if the narrowest statement we can reasonably make is that an applicant's score could be any value between l and h inclusive, where l and h represent the low and high endpoints of the range, then what we are actually saying is that the score is all values between l and h inclusive, because none of those values can be excluded. Thus, in consequence of the large uncertainty about the true values of x that arises from the low-information content of the data available for review, Ordinal 3, for example, translates, from ordinal data to interval data, not to a single point or value, but to a score- set, ranging from 2 to 4 inclusive. Thus, to calculate Nature's Way's aggregate score-set using Dr. Cornew's method, as an example, it is necessary to determine both the applicant's highest possible aggregate score and its lowest possible aggregate score, for these are the endpoints of the range that constitutes the score-set. Finding the high endpoint is accomplished by adding 1 to each Topical Ordinal other than 5, and then computing the aggregate score-set using the mathematical operations described in paragraphs 74 and 75. The following WMTOs (Topical subtotals) are obtained thereby: Cultivation, 1.2250; Processing, 1.0500; Dispensing, 0.6625; MD, 0.2000; and Financials, 0.6667. The high endpoint of Nature's Way's aggregate score-set is the sum of these numbers, or 3.8042.16/ Finding the low endpoint is accomplished roughly in reverse, by subtracting 1 from each Topical Ordinal other than 1, and then computing the aggregate score-set using the mathematical operations described in paragraphs 74 and 75. The low endpoint for Nature's Way works out to 1.9834. Nature's Way's aggregate score-set, thus, is 1.9834-3.8042.17/ This could be written, alternatively, as 2.8938 ± 0.9104 points, or as 2.8938 ± 31.46%. The low and high endpoints of Costa's aggregate score- set are found the same way, and they are, respectively, 3.4000 and 4.8375.18/ Costa's aggregate score-set is 3.4000-4.8375, which could also be written as 4.1188 ± 0.7187 points or 4.1188 ± 17.45%. We can now observe that a score of 2.4000 or more is necessary to satisfy the One Point Condition, and that any score between 2.4000 and 3.8375, inclusive, is both necessary and sufficient to satisfy the One Point Condition. We will call this range (2.4000-3.8375) the Proximity Box. A score outside the Proximity Box on the high end, i.e., a score greater than 3.8375, meets the One Point Condition, of course; however, a score that high, being more than sufficient, is not necessary. Rounding Off the Spurious Digits Remember that the Ordinal 5 does not mean 5 of something that has been counted but the position of 5 in a list of five applicants that have been put in order——nothing more. Recall, too, that there is no interpretable interval between places in a ranking because the difference between 5 and 4 is not the same as that between 4 and 3, etc., and that there is no "second best-and-a-half," which means that taking the average of such numbers is a questionable operation that could easily be misleading if not properly explained. Therefore, as discussed earlier, if the mean of ordinal data is taken, the result must be reported using only as many significant figures as are consistent with the least accurate number, which in this case is one significant figure (whose meaning is only Best, Second Best, Third Best, and so forth). The Department egregiously violated the rule against reliance upon spurious digits, i.e., numbers that lack credible meaning and impart a false sense of accuracy. The Department took advantage of meaningless fractions obtained not by measurement but by mathematical operations, thereby compounding its original error of treating ordinal data as interval data. When the Department says that Nature's Way's aggregate score is 2.8833, it is reporting a number with five significant figures. This number implies that all five figures make sense as increments of a measurement; it implies that the Department's uncertainty about the value is around 0.0001 points——an astonishing degree of accuracy. The trouble is that the aggregate scores, as reported without explanation, are false and deceptive. There is no other way to put it. The Department's reported aggregate scores cannot be rationalized or defended, either, as matters of policy or opinion. This point would be obvious if the Department were saying something more transparent, e.g., that 1 + 1 + 1 + 0 + 0 = 2.8833, for everyone would see the mistake and understand immediately that no policy can change the reality that the sum of three 1s is 3. The falsity at issue is hidden, however, because, to generate each applicant's "aggregate score," the Department started with 42 whole numbers (of ordinal data), each of which is a value from 1 to 5. It then ran the applicant's 42 single- digit, whole number "scores" through a labyrinth of mathematical operations (addition, division, multiplication), none of which improved the accuracy or information content of the original 42 numbers, to produce "aggregate scores" such as 2.8833. This process lent itself nicely to the creation of spreadsheets and tables chocked full of seemingly precise numbers guaranteed to impress.19/ Lacking detailed knowledge (which few people have) about how the numbers were generated, a reasonable person seeing "scores" like 2.8833 points naturally regards them as having substantive value at the microscopic level of ten-thousandths of a point——that's what numbers like that naturally say. He likely believes that these seemingly carefully calibrated measurements are very accurate; after all, results as finely-tuned as 2.8833 are powerful and persuasive when reported with authority. But he has been fooled. The only "measurement" the Department ever took of any applicant was to rank it Best, Second Best, etc.——a "measurement" that was not, and could not have been, fractional. The reported aggregate scores are nothing but weighted averages of ordinal data, dressed up to appear to be something they are not. Remember, the smallest division on the Reviewers' "scale" (using that word loosely here) was 1 rank. No Reviewer used decimal places to evaluate any portion of any application. The aggregate scores implying precision to the ten-thousandth place were all derived from calculations using whole numbers that were code for a value judgment (Best, Second Best, etc.), not quantifiable information. Therefore, in the reported "aggregate scores," none of the digits to the right of first (tenth place) decimal point has any meaning whatsoever; they are nothing but spurious digits introduced by calculations carried out to greater precision than the original data. The first decimal point, moreover, being immediately to the right of the one (and only) significant figure in the aggregate score, is meaningful (assuming that the arithmetic mean of ordinal data even has interpretable meaning, which is controversial) only as an approximation of 1 (whole) rank. Because there is no meaningful fractional rank, the first decimal must be rounded off to avoid a misrepresentation of the data. Ultimately, the only meaning that can be gleaned from the "aggregate score" of 2.8833 is that Nature's Way's typical (or mean) weighted ranking is 2.8833. Because there is no ranking equivalent to 2.8833, this number, if sense is to be made of it, must be rounded to the nearest ranking, which is 3 (because 2.8 ˜ 3), or Third Best. To report this number as if it means something more than that is to mislead. To make decisions based on the premise that 0.8833 means something other than "approximately one whole place in the ranking" is, literally, irrational——indeed, the Department's insistence that its aggregate scores represent true and meaningful quantities of interval data is equivalent, as a statement of logic, to proclaiming that 1 + 1 = 3, the only difference being that the latter statement is immediately recognizable as a delusion. An applicant could only be ranked 1, 2, 3, 4, or 5——not 2.8833 or 4.4000. Likewise, the only meaning that can be taken from the "aggregate score" of 4.4000 is that Costa's average weighted ranking is 4.4000, a number which, for reasons discussed, to be properly understood, must be rounded to the nearest ranking, i.e., 4. The fraction, four-tenths, representing less than half of a position in the ranking, cannot be counted as approximately one whole (additional) place (because 4.4 ? 5). And to treat 0.4000 as meaning four-tenths of a place better than Second Best is absurd. There is no mathematical operation in existence that can turn a number which signifies where in order something is, into one that counts how much of that thing we have. To eliminate the false precision, the spurious digits must be rounded off, which is the established mathematical approach to dealing with numbers that contain uncertainty, as Dr. Cornew credibly confirmed. Rounding to the nearest integer value removes the meaningless figures and eliminates the overprecision manifested by those digits.
The Issue Whether the Respondent, the Department of Business and Professional Regulation, acted in a fraudulent, dishonest, illegal or arbitrary fashion in its determination to award to the Intervenor, Three Oaks Plaza, Ltd., lease number 790:0056.
Findings Of Fact General The Petitioner is Courtyard Plaza, Inc., which is the owner of Courtyard Plaza. The Intervenor is Three Oaks Plaza, Ltd., which is the owner of Three Oaks Plaza. The Respondent is the Department of Business and Professional Regulation, which is a state agency created under Chapter 20, Florida Statutes, and an agency within the context of Chapter 255, Florida Statutes. Requirements of the RFP Respondent advertised an RFP pertaining to the provision of lease premises for lease number 790:0056 in a defined geographical area within Jacksonville, Florida. (T. 214). The geographical area for the RFP was approved by the Division of Facilities Management of the Department of Management Services. (Tx. 250, 256). The requirement of the RFP were as follows: Provision of 15,674 square feet of general office space located in Duval County. (A map accompanied the RFP which further limited the location of the lease to an area roughly bounded by Arlington Expressway to the north, Southside Boulevard to the east, Baymeadows Road to the south, and Phillips Highway-University Boulevard, and the St. Johns River to the west.); Availability of the space by December 1, 1995, for a period of five years with an option to renew for 10/1 years; Full service utilities, interior and exterior maintenance, janitorial services and supplies; Availability of 72 off-street [parking] spaces within 150 feet of the facility's main entrance dedicated to the exclusive use of employees and clients of the Department; Availability to public transportation; Dining facilities within 2 blocks of the offered facilities. Building code minimum of 50 pounds per square foot liveload, with 100 pounds per square foot live-load available in some sections to house vaults (applicable to multistory buildings); Departmental approval of renovation plans and specifications; and As and for security, locks on all outside/interior office doors, locks on all outside windows, a night light located within 10 feet of any outside door and night lights in all parking areas nearest the building. Pre-bid Procedures Petitioner and Intervenor attended a pre-bid conference held by the Respondent on February 28, 1995. Most of the evaluators were at the pre-bid conference. They were Brad Engleman, Tracy Pyke, Bob Miller, Elizabeth Doyle, Jerry Jenson and Jim Bob Cooper. Other than Engleman and Pyke, who were stationed in Tallahassee, the evaluators served as division chiefs in Jacksonville. There were three proposals submitted in response to the RFP: one from Three Oaks, one from Courtyard, and one from the current landlord, Adnan El- Yazingi. A review by Engleman and Pyke determined that El-Yazingi's proposal was not responsive on the basis that it was outside the geographic area designated in the RFP. On March 29, 1995, the evaluators visited the facilities proposed by Courtyard and Three Oaks during the day. The evaluators visited Courtyard first. Evaluation Score Sheets The evaluators used score sheets provided by the Department to evaluate the facilities. The evaluators did not see the bid specifications, and worked from the criteria of the evaluation sheets which do not relate to the criteria in the RFP. The score sheets for each facility contained the following criteria: Rental Rates (a) [39 Points] Based upon total present value for basic term of the lease using present value discount rate. [1 Point] For optional renewal terms of lease. Location: Environmental factors, including the physical characteristics of the building and the area surround it, efficiency and economic operations in the requested space. (a) [5 Points] Frequency and availability of satisfactory public transportation within proximity of the offered space. [10 Points] Proximity of offered space to clients to be served by the Department at this facility. [10 Points] Aesthetics of building, property the building sits on, and the surrounding neighborhood. [15 Points] Security issues posed by building, by associated parking and surrounding neighborhood. The criteria used for the security issue of the neighborhood shall be determined by the agency. Police crime statistics may be incorporated into the criteria. Facility (a) [10 Points] Susceptibility of the design of the space offered to efficient layout and good utilization, i.e., ability to house large units together and in close proximity to interdependent units. [10 Points] Susceptibility of the building, parking area and property as a whole for future expansions. Evaluation The evaluation score sheets were divided into two parts, the first part dealing with the financial criteria and involving a computation and comparison of the present value of the lease proposal of each bidder, and the second part assessing if an option period were offered. The first part was worth 39 points, and the second was worth one (1) point. The most cost effective bid received 39 points, and that value of that bid became the denominator of a fraction in which the value of the other proposal(s) became the numerator. The total number of available points (39) was then multiplied by the aforesaid fraction to complete the proposal(s). If the bidder provided for additional option period, the bidder received an additional one point. Because the computation of the present value was a mechanical act in which there was no discretion used by the evaluators, it was computed by Engleman and Pyke in Tallahassee after the evaluators completed the site visits and assessments of each facility. The present value of each of the proposals was computed using a computer program provided by Management Services. Courtyard was determined to be the lowest and best bid, and received 39 points. The value of the Three Oaks was also computed, and Three Oaks received 32 points. Both bidders provided for options. Three Oaks received 33 points for it financial portion of the bid, and Courtyard received 40 points. On Site Evaluations The evaluators based their scoring of the score sheets solely on the on-site visits, and nothing else. 2(a) Availability of Public Transportation [5 points] The record reveals that public bus transportation is available on a regularly scheduled basis within one half block of both Courtyard and Three Oaks. The evaluators, whose initials are indicated below, scored element 2(a) as follows: B.E. B.M. J.J. J.C. E.D. T.P. Ctyd 5 4 4 4 3 5 Oaks 5 5 4 4 5 5 2(b) Proximity to Clients [10 points] The Courtyard in the RFP is located almost in the middle of the geographic area designated by the Department. It is on Beach Boulevard two miles east of the Southside Boulevard at the intersection of Beach Boulevard and Parental Home Road. The Courtyard is approximately one-half block from the Beach Boulevard access to the Commodore Point Expressway which runs from Beach Boulevard northwest to the vicinity of the Gator Bowl where it splits and runs due west downtown to the vicinity of city-county buildings downtown, and due north to intersect with the accesses to the Mathews Bridge and the Haines Expressway. Three Oaks is located on the most northerly boundary of the geographic area designated by the RFP. Three Oaks is located on the south side of Arlington Expressway, approximately three miles east the Mathews Bridge and two miles west of the intersection of Southside Boulevard and Arlington Expressway. From the western end of the Mathews Bridge it is approximately two miles from I-95 via high-speed routes. Southside Boulevard runs south to intersect with I-295/I-95, and north to cross the St. Johns River over the Dames Point Bridge. The evaluators scored element 2(b) as follows: B.E. B.M. J.J. J.C. E.D. T.P. Ctyd 6 5 8 6 3 4 Oaks 10 10 8 6 7 8 2(c) Aesthetics of the Building [10 points] The Courtyard property is a medium sized office park, consisting of two, two story buildings, each with two wings joined at a ninety degree angle at main entrance area. The site visit to the Courtyard was conducted by Mary Farwell, President of Petra Management, Inc., the agent for the Petitioner, Courtyard. Ms. Farwell explained that all the buildings had been vacant for the past two years and the landscaping and the buildings had not been kept up; however, that the landlord was committed to making improvements to the landscaping and buildings to suit the tenants. She made a presentation in which she showed the evaluators various architectural renderings of modifications to the exterior elevations of the buildings which Courtyard was willing to undertake for a major lessor who would be able to provide input to the design. The Courtyard site had many mature trees, and shaded areas around the building and parking lots. The parking lots needed maintenance, resealing, and restriping. Ms. Farwell also pointed out an area away from the building being proposed, but on the property, where flooding had occurred during the recent serious storms. She told the evaluators that the Department of Transportation was committed to changing the drainage system which adjoined the property to correct the problem. Three Oaks proposed space split between two floors in a seven story office building which is one of three similar buildings located on the property. The part of the space being offered was not completed, but the evaluators were shown existing completed and occupied space in the building. It was represented that the proposed space would be completed in a comparable fashion. The buildings at Three Oaks were landscaped immediately adjacent to the buildings, but were surrounded by large paved parking lots devoid of trees or landscaping. The evaluators scored element 2(c) as follows: B.E. B.M. J.J. J.C. E.D. T.P. Ctyd 7 4 8 6 5 6 Oaks 9 9 9 10 9 9 2(d) Security Issues [15 points] There were night lights at Courtyard as required by the RFP. The parking lots at The Courtyard were lit with lights mounted on the buildings as required. Because of the growth of the trees surrounding the buildings, these lights did not fully illuminate the parking lot. This was an oft mentioned concern of the evaluators; however, this problem was capable of being altered by the addition of lights and pruning of trees and other on-site landscaping which the landlord was committed to do. It was the impression of the evaluators that The Courtyard was in a "worse" neighborhood that Three Oaks. These concerns were not mentioned to the landlord's representative, and crime statistics were not obtained from the local police regarding comparative crime rates. The photographic evidence presented by both sides of The Courtyard reveals no vandalism or graffiti on buildings vacant for two years. The evaluators ratings of Courtyard on this requirement were highly subjective and clearly beyond the requirements of the RFP. The open parking lots at Three Oaks were lighted, and the evaluators gave great value to the representation that there were private security officers available to assist occupants to their cars. The fact that a bank with its security personnel was located in the building was also brought to the attention of the evaluators. Security personnel were not RFP requirements. The evaluators scored element 2 (d) as follows: B.E. B.M. J.J. J.C. E.D. T.P. Ctyd 10 7 10 7 5 10 Oaks 13 13 11 12 15 13 3(a) Efficient Layout [10 points] Ms. Farwell explained to the evaluators that the Courtyard's landlord was prepared to gut the two story wing making up one-half of the eastern building which was being proposed for their use, and remodel the interior space to the Department's specifications. Although, some of the activities would have had to be separated on the two floor, this was not an operational problem for the activities which were geographically separated at the time. Ms. Farwell also indicated there had been discussions about providing day care services on site through a local agency. Three Oaks proposed to construct the interior space to the Department's specification. There were already some elements of the Department located in the building, and this would have been a small benefit for the Department, although the Department's units were independent and used to being geographically separated. Both bidders could provide spaces meeting the vault floor load requirements. Both bidders could provide the requisite number of parking spaces within the required distance. The evaluators scored element 3 (a) as follows: B.E. B.M. J.J. J.C. E.D. T.P. Ctyd 9 8 9 10 9 10 Oaks 10 9 9 9 9 10 Capacity to expand space and parking [10 points] The Courtyard had three additional unrented buildings the same size as the one presented in its proposal. As the major lessor, the Department would have priority in expanding into the other buildings if required. Parking as required was available and adequate for any future expansion. Three Oaks would have had to move other tenants to accommodate the further expansion of the lease; however, the landlord was willing to move other tenants who were on short terms leases. Parking as required was available and adequate for any future expansion. Both bidders had the office and parking capacity to accommodate further expansion. The evaluators scored element 3 (b) as follows: B.E. B.M. J.J. J.C. E.D. T.P. Ctyd 8 5 8 9 8 9 Oaks 9 6 8 9 9 10 Bid Criteria from RFP not addressed on the score sheet Both facilities had dinning facilities within two blocks. Neither landlords had completed the facilities offered; however, the RFP did not envision completed facilities because it called for Departmental approval of renovation plans and specifications. Both landlords offered to make their facilities available before December 1, 1995, and the nature of the proposed renovations at both facilities was such that this could be done. No evidence was received specifically on the nature and quality of the janitorial supplies and services; however, both landlords offered to provide these supplies and services together with maintenance on the facilities. Because The Courtyard was unoccupied, the evaluators could not assess the adequacy of the services at that facility. They did evaluate favorably the maintenance and upkeep at Three Oaks. Similarly, no evidence was received specifically on locks on the doors and windows at the facilities; however, both facilities had fixed windows, and the doors were capable of locking. There was no evidence of fraudulent activity in the exercise of the Respondent's discretion. There is no evidence of dishonesty in the exercise of Respondent's discretion. There is no evidence of illegal activity in the exercise of the Respondent's discretion. There is evidence that the Respondent's evaluations considered additional factors in assessing location and security of the properties in evaluating the proposals which were not part of the criteria of the RFP.
Recommendation Based upon the foregoing Findings of Fact and Conclusions of Law set forth herein, it is, RECOMMENDED: That the Department not award the contract, and reinitiate the evaluation process using criteria clearly reflective of the requirements stated in the RFP. DONE and ENTERED this 7th day of July, 1995, in Tallahassee, Florida. STEPHEN F. DEAN, Hearing Officer Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-1550 (904) 488-9675 Filed with the Clerk of the Division of Administrative Hearings this 7th day of July, 1995. APPENDIX The parties filed proposed findings of fact which were read and considered. The following states which of those findings were adopted, and which were rejected and why: Petitioner's Recommended Order Findings Paragraphs 1,2 Paragraph 9 Paragraph 3 Rejected as contrary to most credible evidence. Paragraph 4 Subsumed in Paragraph 9 et seq. Paragraph 5a Subsumed in Paragraphs 10,11. Paragraph 5b Subsumed in Paragraph 15 et seq. Paragraph 5c Subsumed in Paragraph 19 et seq. which are based upon the more credible evidence. Paragraph 5d Subsumed in Paragraph 26 et seq. Paragraph 6a Subsumed in Paragraphs 13,14. Paragraph 6b Subsumed in Paragraph 15 et seq. Paragraph 6c Subsumed in Paragraph 19 et seq. Paragraph 6d Subsumed in Paragraph 16 et seq. Paragraph 7 Subsumed in Paragraphs 13-29. Paragraphs 8,9 Rejected as Conclusion of Law. Respondent's Recommended Order Findings Paragraphs 1-4 Paragraph 1-4 Paragraph 5 Paragraph 6,7-11 Paragraph 6 Paragraphs 14,18,25,29,33,37,38. Paragraph 7 Statement of Case. Paragraph 8 Paragraph 39. Paragraphs 9-11 Paragraph 42. Paragraphs 12-14 Rejected as contrary to best evidence. Paragraphs 15,16 Irrelevant. Paragraph 17 Subsumed in 34. Paragraphs 18-21 Subsumed in 10,11. Paragraph 22 Paragraphs 9,12. Paragraph 23 Paragraph 13. Paragraph 24 Subsumed in 16,17. Paragraphs 25,26 Subsumed in 15,16. Paragraph 27 There was testimony to this effect which indicates that the evaluators considered issues which were not part of the RFP. Paragraph 28 Rejected as contrary to best evidence. See discussion in Paragraph 53. Paragraph 29 True, but this was not part of the criteria stated in the RFP. Paragraphs 30-33 Subsumed in Paragraph 30 et seq. Paragraphs 34-36 Subsumed in Paragraph 26 et seq. Paragraph 37 Subsumed in Paragraph 30 et seq. Paragraphs 38,39 Subsumed in Paragraph 34 et seq. Paragraph 40 Paragraphs 34,35. Paragraph 41 Paragraph 22. Paragraph 42 Subsumed in specific findings addressed above. Intervenor's Recommended Order Findings Paragraphs 1,2 Paragraph 4. Paragraph 3 Paragraph 7. Paragraph 4 Paragraph 6. Paragraph 5 Paragraph 8-11. Paragraph 5(Sic) Paragraph 8,12. Paragraph 6 Subsumed in Paragraphs 11,12,38. Paragraph 7 Irrelevant. Paragraph 8 Subsumed in Paragraph 38. Paragraph 9 Adopted in part and rejected in part. See specific findings in paragraphs 1-42. Paragraph 10 Conclusion of Law. COPIES FURNISHED: Mary C. Sorrell, Esquire 2275 Atlantic Blvd. Jacksonville, FL 32266 William M. Woodyard, Esquire Stephen Willis, Esquire Department of Business and Professional Regulation 1940 North Monroe Street Northwood Centre Tallahassee, FL 32399-0750 Melissa Fletcher Allaman, Esquire Thomas M. Ervin, Jr., Esquire Ervin, Varn, Jacobs, Odom & Ervin 305 South Gadsden Street Post Office Drawer 1170 Tallahassee, FL 32302 Delane Anderson, Acting Secretary Department of Business and Professional Regulation 1940 North Monroe Street Northwood Centre Tallahassee, FL 32399-0750 Jack McRay, General Counsel Department of Business and Professional Regulation 1940 North Monroe Street Northwood Centre Tallahassee, FL 32399-0750
The Issue The issue to be resolved in this proceeding concerns whether the Department of Revenue (Department, DOR) acted clearly erroneously, contrary to competition, arbitrarily or capriciously when it evaluated the Petitioner's submittal in response to an Invitation to Negotiate (ITN) for a child support enforcement automated management system-compliance enforcement (CAMS CE) in which it awarded the Petitioner a score of 140 points out of a possible 230 points and disqualified the Petitioner from further consideration in the invitation to negotiate process.
Findings Of Fact Procurement Background: The Respondent, the (DOR) is a state agency charged with the responsibility of administering the Child Support Enforcement Program (CSE) for the State of Florida, in accordance with Section 20.21(h), Florida Statutes. The DOR issued an ITN for the CAMS Compliance Enforcement implementation on February 1, 2002. This procurement is designed to give the Department a "state of the art system" that will meet all Federal and State Regulations and Policies for Child Support Enforcement, improve the effectiveness of collections of child support and automate enforcement to the greatest extent possible. It will automate data processing and other decision- support functions and allow rapid implementation of changes in regulatory requirements resulting from revised Federal and State Regulation Policies and Florida initiatives, including statutory initiatives. CSE services suffer from dependence on an inadequate computer system known as the "FLORIDA System" which was not originally designed for CSE and is housed and administered in another agency. The current FLORIDA System cannot meet the Respondent's needs for automation and does not provide the Respondent's need for management and reporting requirements and the need for a more flexible system. The DOR needs a system that will ensure the integrity of its data, will allow the Respondent to consolidate some of the "stand-alone" systems it currently has in place to remedy certain deficiencies of the FLORIDA System and which will help the Child Support Enforcement system and program secure needed improvements. The CSE is also governed by Federal Policy, Rules and Reporting requirements concerning performance. In order to improve its effectiveness in responding to its business partners in the court system, the Department of Children and Family Services, the Sheriff's Departments, employers, financial institutions and workforce development boards, as well as to the Federal requirements, it has become apparent that the CSE agency and system needs a new computer system with the flexibility to respond to the complete requirements of the CSE system. In order to accomplish its goal of acquiring a new computer system, the CSE began the procurement process. The Department hired a team from the Northrup Grumman Corporation headed by Dr. Edward Addy to head the procurement development process. Dr. Addy began a process of defining CSE needs and then developing an ITN which reflected those needs. The process included many individuals in CSE who would be the daily users of the new system. These individuals included Andrew Michael Ellis, Revenue Program Administrator III for Child Support Enforcement Compliance Enforcement; Frank Doolittle, Process Manager for Child Support Enforcement Compliance Enforcement and Harold Bankirer, Deputy Program Director for the Child Support Enforcement Program. There are two alternative strategies for implementing a large computer system such as CAMS CE: a customized system developed especially for CSE or a Commercial Off The Shelf, Enterprise Resource Plan (COTS/ERP). A COTS/ERP system is a pre-packaged software program, which is implemented as a system- wide solution. Because there is no existing COTS/ERP for child support programs, the team recognized that customization would be required to make the product fit its intended use. The team recognized that other system attributes were also important, such as the ability to convert "legacy data" and to address such factors as data base complexity and data base size. The Evaluation Process: The CAMS CE ITN put forth a tiered process for selecting vendors for negotiation. The first tier involved an evaluation of key proposal topics. The key topics were the vendors past corporate experience (past projects) and its key staff. A vendor was required to score 150 out of a possible 230 points to enable it to continue to the next stage or tier of consideration in the procurement process. The evaluation team wanted to remove vendors who did not have a serious chance of becoming the selected vendor at an early stage. This would prevent an unnecessary expenditure of time and resources by both the CSE and the vendor. The ITN required that the vendors provide three corporate references showing their past corporate experience for evaluation. In other words, the references involved past jobs they had done for other entities which showed relevant experience in relation to the ITN specifications. The Department provided forms to the vendors who in turn provided them to their corporate references that they themselves selected. The vendors also included a summary of their corporate experience in their proposal drafted by the vendors themselves. Table 8.2 of the ITN provided positive and negative criteria by which the corporate references would be evaluated. The list in Table 8.2 is not meant to be exhaustive and is in the nature of an "included but not limited to" standard. The vendors had the freedom to select references whose projects the vendors' believed best fit the criteria upon which each proposal was to be evaluated. For the key staff evaluation standard, the vendors provided summary sheets as well as résumés for each person filling a lead role as key staff members on their proposed project team. Having a competent project team was deemed by the Department to be critical to the success of the procurement and implementation of a large project such as the CAMS CE. Table 8.2 of the ITN provided the criteria by which the key staff would be evaluated. The Evaluation Team: The CSE selected an evaluation team which included Dr. Addy, Mr. Ellis, Mr. Bankirer, Mr. Doolittle and Mr. Esser. Although Dr. Addy had not previously performed the role of an evaluator, he has responded to several procurements for Florida government agencies. He is familiar with Florida's procurement process and has a doctorate in Computer Science as well as seventeen years of experience in information technology. Dr. Addy was the leader of the Northrup Grumman team which primarily developed the ITN with the assistance of personnel from the CSE program itself. Mr. Ellis, Mr. Bankirer and Mr. Doolittle participated in the development of the ITN as well. Mr. Bankirer and Mr. Doolittle had previously been evaluators in other procurements for Federal and State agencies prior to joining the CSE program. Mr. Esser is the Chief of the Bureau of Information Technology at the Department of Highway Safety and Motor Vehicles and has experience in similar, large computer system procurements at that agency. The evaluation team selected by the Department thus has extensive experience in computer technology, as well as knowledge of the requirements of the subject system. The Department provided training regarding the evaluation process to the evaluators as well as a copy of the ITN, the Source Selection Plan and the Source Selection Team Reference Guide. Section 6 of the Source Selection Team Reference Guide entitled "Scoring Concepts" provided guidance to the evaluators for scoring proposals. Section 6.1 entitled "Proposal Evaluation Specification in ITN Section 8" states: Section 8 of the ITN describes the method by which proposals will be evaluated and scored. SST evaluators should be consistent with the method described in the ITN, and the source selection process documented in the Reference Guide and the SST tools are designed to implement this method. All topics that are assigned to an SST evaluator should receive at the proper time an integer score between 0 and 10 (inclusive). Each topic is also assigned a weight factor that is multiplied by the given score in order to place a greater or lesser emphasis on specific topics. (The PES workbook is already set to perform this multiplication upon entry of the score.) Tables 8-2 through 8-6 in the ITN Section 8 list the topics by which the proposals will be scored along with the ITN reference and evaluation and scoring criteria for each topic. The ITN reference points to the primary ITN section that describes the topic. The evaluation and scoring criteria list characteristics that should be used to affect the score negatively or positively. While these characteristics should be used by each SST evaluator, each evaluator is free to emphasize each characteristic more or less than any other characteristic. In addition, the characteristics are not meant to be inclusive, and evaluators may consider other characteristics that are not listed . . . (Emphasis supplied). The preponderant evidence demonstrates that all the evaluators followed these instructions in conducting their evaluations and none used a criterion that was not contained in the ITN, either expressly or implicitly. Scoring Method: The ITN used a 0 to 10 scoring system. The Source Selection Team Guide required that the evaluators use whole integer scores. They were not required to start at "7," which was the average score necessary to achieve a passing 150 points, and then to score up or down from 7. The Department also did not provide guidance to the evaluators regarding a relative value of any score, i.e., what is a "5" as opposed to a "6" or a "7." There is no provision in the ITN which establishes a baseline score or starting point from which the evaluators were required to adjust their scores. The procurement development team had decided to give very little structure to the evaluators as they wanted to have each evaluator score based upon his or her understanding of what was in the proposal. Within the ITN the development team could not sufficiently characterize every potential requirement, in the form that it might be submitted, and provide the consistency of scoring that one would want in a competitive environment. This open-ended approach is a customary method of scoring, particularly in more complex procurements in which generally less guidance is given to evaluators. Providing precise guidance regarding the relative value of any score, regarding the imposition of a baseline score or starting point, from which evaluators were required to adjust their scores, instruction as to weighing of scores and other indicia of precise structure to the evaluators would be more appropriate where the evaluators themselves were not sophisticated, trained and experienced in the type of computer system desired and in the field of information technology and data retrieval generally. The evaluation team, however, was shown to be experienced and trained in information technology and data retrieval and experienced in complex computer system procurement. Mr. Barker is the former Bureau Chief of Procurement for the Department of Management Services. He has 34 years of procurement experience and has participated in many procurements for technology systems similar to CAMS CE. He established that the scoring system used by the Department at this initial stage of the procurement process is a common method. It is customary to leave the numerical value of scores to the discretion of the evaluators based upon each evaluator's experience and review of the relevant documents. According wider discretion to evaluators in such a complex procurement process tends to produce more objective scores. The evaluators scored past corporate experience (references) and key staff according to the criteria in Table 8.2 of the ITN. The evaluators then used different scoring strategies within the discretion accorded to them by the 0 to 10 point scale. Mr. Bankirer established a midrange of 4 to 6 and added or subtracted points based upon how well the proposal addressed the CAMS CE requirements. Evaluator Ellis used 6 as his baseline and added or subtracted points from there. Dr. Addy evaluated the proposals as a composite without a starting point. Mr. Doolittle started with 5 as an average score and then added or subtracted points. Mr. Esser gave points for each attribute in Table 8.2, for key staff, and added the points for the score. For the corporate reference criterion, he subtracted a point for each attribute the reference lacked. As each of the evaluators used the same methodology for the evaluation of each separate vendor's proposal, each vendor was treated the same and thus no specific prejudice to KPMG was demonstrated. Corporate Reference Evaluation: KPMG submitted three corporate references: Duke University Health System (Duke), SSM Health Care (SSM), and Armstrong World Industries (Armstrong). Mr. Bankirer gave the Duke reference a score of 6, the SSM reference a score of 5 and the Armstrong reference a score of 7. Michael Strange, the KPMG Business Development Manager, believed that 6 was a low score. He contended that an average score of 7 was required to make the 150-point threshold for passage to the next level of the ITN consideration. Therefore, a score of 7 would represent minimum compliance, according to Mr. Strange. However, neither the ITN nor the Source Selection Team Guide identified 7 as a minimally compliant score. Mr. Strange's designation of 7 as a minimally compliant score is not provided for in the specifications or the scoring instructions. Mr. James Focht, Senior Manager for KPMG testified that 6 was a low score, based upon the quality of the reference that KPMG had provided. However, Mr. Bankirer found that the Duke reference was actually a small-sized project, with little system development attributes, and that it did not include information regarding a number of records, the data base size involved, the estimated and actual costs and attributes of data base conversion. Mr. Bankirer determined that the Duke reference had little similarity to the CAMS CE procurement requirements and did not provide training or data conversion as attributes for the Duke procurement which are attributes necessary to the CAMS CE procurement. Mr. Strange and Mr. Focht admitted that the Duke reference did not specifically contain the element of data conversion and that under the Table 8.2, omission of this information would negatively affect the score. Mr. Focht admitted that there was no information in the Duke Health reference regarding the number of records and the data base size, all of which factors diminish the quality of Duke as a reference and thus the score accorded to it. Mr. Strange opined that Mr. Bankirer had erred in determining that the Duke project was a significantly small sized project since it only had 1,500 users. Mr. Focht believed that the only size criterion in Table 8.2 was the five million dollar cost threshold, and, because KPMG indicated that the project cost was greater than five million dollars, that KPMG had met the size criterion. Mr. Focht believed that evaluators had difficulty in evaluating the size of the projects in the references due to a lack of training. Mr. Focht was of the view that the evaluator should have been instructed to make "binary choices" on issues such as size. He conceded, however, that evaluators may have looked at other criteria in Table 8.2 to determine the size of the project, such as database size and number of users. However, the corporate references were composite scores by the evaluators, as the ITN did not require separate scores for each factor in Table 8.2. Therefore, Mr. Focht's focus on binary scoring for size, to the exclusion of other criteria, mis-stated the objective of the scoring process. The score given to the corporate references was a composite of all of the factors in Table 8.2, and not merely monetary value size. Although KPMG apparently contends that size, in terms of dollar value, is the critical factor in determining the score for a corporate reference, the vendor questions and answers provided at the pre-proposal conference addressed the issue of relevant criteria. Question 40 of the vendor questions and answers, Volume II, did not single out "project greater than five million dollars" as the only size factor or criterion. QUESTION: Does the state require that each reference provided by the bidder have a contract value greater than $5 million; and serve a large number of users; and include data conversion from a legacy system; and include training development? ANSWER: To get a maximum score for past corporate experience, each reference must meet these criteria. If the criteria are not fully met, the reference will be evaluated, but will be assigned a lower score depending upon the degree to which the referenced project falls short of these required characteristics. Therefore, the cost of the project is shown to be only one component of a composite score. Mr. Strange opined that Mr. Bankirer's comment regarding the Duke reference, "little development, mostly SAP implementation" was irrelevant. Mr. Strange's view was that the CAMS CE was not a development project and Table 8.2 did not specifically list development as a factor on which proposals would be evaluated. Mr. Focht stated that in his belief Mr. Bankirer's comment suggested that Mr. Bankirer did not understand the link between the qualifications in the reference and the nature of KPMG's proposal. Both Strange and Focht believe that the ITN called for a COTS/ERP solution. Mr. Focht stated that the ITN references a COTS/ERP approach numerous times. Although many of the references to COTS/ERP in the ITN also refer to development, Mr. Strange also admitted that the ITN was open to a number of approaches. Furthermore, both the ITN and the Source Selection Team Guide stated that the items in Table 8.2 are not all inclusive and that the evaluators may look to other factors in the ITN. Mr. Bankirer noted that there is no current CSE COTS/ERP product on the market. Therefore, some development will be required to adapt an off-the-shelf product to its intended use as a child support case management system. Mr. Bankirer testified that the Duke project was a small-size project with little development. Duke has three sites while CSE has over 150 sites. Therefore, the Duke project is smaller than CAMS. There was no information provided in the KPMG submittal regarding data base size and number of records with regard to the Duke project. Mr. Bankirer did not receive the information he needed to infer a larger sized-project from the Duke reference. Mr. Esser also gave the Duke reference a score of 6. The reference did not provide the data base information required, which was the number of records in the data base and the number of "gigabytes" of disc storage to store the data, and there was no element of legacy conversion. Dr. Addy gave the Duke reference a score of 5. He accepted the dollar value as greater than five million dollars. He thought that the Duke Project may have included some data conversion, but it was not explicitly stated. The Duke customer evaluated training so he presumed training was provided with the Duke project. The customer ratings for Duke were high as he expected they would be, but similarity to the CAMS CE system was not well explained. He looked at size in terms of numbers of users, number of records and database size. The numbers that were listed were for a relatively small-sized project. There was not much description of the methodology used and so he gave it an overall score of 5. Mr. Doolittle gave the Duke reference a score of 6. He felt that it was an average response. He listed the number of users, the number of locations, that it was on time and on budget, but found that there was no mention of data conversion, database size or number of records. (Consistent with the other evaluators). A review of the evaluators comments makes it apparent that KPMG scores are more a product of a paucity of information provided by KPMG corporate references instead of a lack of evaluator knowledge of the material being evaluated. Mr. Ellis gave a score of 6 for the Duke reference. He used 6 as his baseline. He found the required elements but nothing more justifying in his mind raising the score above 6. Mr. Focht and Mr. Strange expressed the same concerns regarding Bankirer's comment, regarding little development, for the SSM Healthcare reference as they had for the Duke Health reference. However, both Mr. Strange and Mr. Focht admitted that the reference provided no information regarding training. Mr. Strange admitted that the reference had no information regarding data conversion. Training and data conversion are criteria contained in Table 8.2. Mr. Strange also admitted that KPMG had access to Table 8.2 before the proposal was submitted and could have included the information in the proposal. Mr. Bankirer gave the SSM reference a score of 5. He commented that the SAP implementation was not relevant to what the Department was attempting to do with the CAMS CE system. CAMS CE does not have any materials management or procurement components, which was the function of the SAP components and the SSM reference procurement or project. Additionally, there was no training indicated in the SSM reference. Mr. Esser gave the SSM reference a score of 3. His comments were "no training provided, no legacy data conversion, project evaluation was primarily for SAP not KPMG". However, it was KPMG's responsibility in responding to the ITN to provide project information concerning a corporate reference in a clear manner rather than requiring that an evaluator infer compliance with the specifications. Mr. Focht believed that legacy data conversion could be inferred from the reference's description of the project. Mr. Strange opined that Mr. Esser's comment was inaccurate as KPMG installed SAP and made the software work. Mr. Esser gave the SSM reference a score of 3 because the reference described SAP's role, but not KPMG's role in the installation of the software. When providing information in the reference SSM gave answers relating to SAP to the questions regarding system capability, system usability, system reliability but did not state KPMG's role in the installation. SAP is a large enterprise software package. This answer created an impression of little KPMG involvement in the project. Dr. Addy gave the SSM reference a score of 6. Dr. Addy found that the size was over five million dollars and customer ratings were high except for a 7 for usability with reference to a "long learning curve" for users. Data conversion was implied. There was no strong explanation of similarity to CAMS CE. It was generally a small-sized project. He could reason some similarity into it, even though it was not well described in the submittal. Mr. Doolittle gave the SSM reference a score of 6. Mr. Doolittle noted, as positive factors, that the total cost of the project was greater than five million dollars, that it supported 24 sites and 1,500 users as well "migration from a mainframe." However, there were negative factors such as training not being mentioned and a long learning curve for its users. Mr. Ellis gave a score of 6 for SSM, feeling that KPMG met all of the requirements but did not offer more than the basic requirements. Mr. Strange opined that Mr. Bankirer, Dr. Addy and Mr. Ellis (evaluators 1, 5 and 4) were inconsistent with each other in their evaluation of the SSM reference. He stated that this inconsistency showed a flaw in the evaluation process in that the evaluators did not have enough training to uniformly evaluate past corporate experience, thereby, in his view, creating an arbitrary evaluation process. Mr. Bankirer gave the SSM reference a score of 5, Ellis a score of 6, and Addy a score of 6. Even though the scores were similar, Mr. Strange contended that they gave conflicting comments regarding the size of the project. Mr. Ellis stated that the size of the project was hard to determine as the cost was listed as greater than five million dollars and the database size given, but the number of records was not given. Mr. Bankirer found that the project was low in cost and Dr. Addy stated that over five million dollars was a positive factor in his consideration. However, the evaluators looked at all of the factors in Table 8.2 in scoring each reference. Other factors that detracted from KPMG's score for the SSM reference were: similarity to the CAMS system not being explained, according to Dr. Addy; no indication of training (all of the evaluators); the number of records not being provided (evaluator Ellis); little development shown (Bankirer) and usability problems (Dr. Addy). Mr. Strange admitted that the evaluators may have been looking at other factors besides the dollar value size in order to score the SSM reference. Mr. Esser gave the Armstrong reference a score of 6. He felt that the reference did not contain any database information or cost data and that there was no legacy conversion shown. Dr. Addy also gave Armstrong a score of 6. He inferred that this reference had data conversion as well as training and the high dollar volume which were all positive factors. He could not tell, however, from the project description, what role KPMG actually had in the project. Mr. Ellis gave a score of 7 for the Armstrong reference stating that the Armstrong reference offered more information regarding the nature of the project than had the SSM and Duke references. Mr. Bankirer gave KPMG a score of 7 for the Armstrong reference. He found that the positive factors were that the reference had more site locations and offered training but, on the negative side, was not specific regarding KPMG's role in the project. Mr. Focht opined that the evaluators did not understand the nature of the product and services the Department was seeking to obtain as the Department's training did not cover the nature of the procurement and the products and services DOR was seeking. However, when he made this statement he admitted he did not know the evaluators' backgrounds. In fact, Bankirer, Ellis, Addy and Doolittle were part of a group that developed the ITN and clearly knew what CSE was seeking to procure. Further, Mr. Esser stated that he was familiar with COTS and described it as a commercial off-the-shelf software package. Mr. Esser explained that an ERP solution or Enterprise Resource Plan is a package that is designed to do a series of tasks, such as produce standard reports and perform standard operations. He did not believe that he needed further training in COTS/ERP to evaluate the proposals. Mr. Doolittle was also familiar with COTS/ERP and believed, based on the amount of funding, that it was a likely response to the ITN. Dr. Addy's doctoral dissertation research was in the area of software re-use. COTS is one of the components that comprise a development activity and re-use. He became aware during his research of how COTS packages are used in software engineering. He has also been exposed to ERP packages. ERP is only one form of a COTS package. In regard to the development of the ITN and the expectations of the development team, Dr. Addy stated that they were amenable to any solution that met the requirements of the ITN. They fully expected the compliance solutions were going to be comprised of mostly COTS and ERP packages. Furthermore, the ITN in Section 1.1, on page 1-2 states, ". . . FDOR will consider an applicable Enterprise Resource Planning (ERP) or Commercial Off the Shelf (COTS) based solution in addition to custom development." Clearly, this ITN was an open procurement and to train evaluators on only one of the alternative solutions would have biased the evaluation process. Mr. Doolittle gave each of the KPMG corporate references a score of 6. Mr. Strange and Mr. Focht questioned the appropriateness of these scores as the corporate references themselves gave KPMG average ratings of 8.3, 8.2 and 8.0. However, Mr. Focht admitted that Mr. Doolittle's comments regarding the corporate references were a mixture of positive and negative comments. Mr. Focht believed, however, that as the reference corporations considered the same factors for providing ratings on the reference forms, that it was inconsistent for Mr. Doolittle to separately evaluate the same factors that the corporations had already rated. However, there is no evidence in the record that KPMG provided Table 8.2 to the companies completing the reference forms and that the companies consulted the table when completing their reference forms. Therefore, KPMG did not prove that it had taken all measures available to it to improve its scores. Moreover, Mr. Focht's criticism would impose a requirement on Mr. Doolittle's evaluation which was not supported by the ITN. Mr. Focht admitted that there was no criteria in the ITN which limited the evaluator's discretion in scoring to the ratings given to the corporate references by those corporate reference customers. All of the evaluators used Table 8.2 as their guide for scoring the corporate references. As part of his evaluation, Dr. Addy looked at the methodology used by the proposers in each of the corporate references to implement the solution for that reference company. He was looking at methodology to determine its degree of similarity to CAMS CE. While not specifically listed in Table 8.2 as a similarity to CAMS, Table 8.2 states that the list is not all inclusive. Clearly, methodology is a measure of similarity and therefore is not an arbitrary criterion. Moreover, as Dr. Addy used the same process and criteria in evaluating all of the proposals there was no prejudice to KPMG by use of this criterion since all vendors were subjected to it. Mr. Strange stated that KPMG appeared to receive lower scores for SAP applications than other vendors. For example, evaluator 1 gave a score of 7 to Deloitte's reference for Suntax. Suntax is an SAP implementation. It is difficult to draw comparisons across vendors, yet the evaluators consistently found that KPMG references lacked key elements such as data conversion, information on starting and ending costs, and information on database size. All of these missing elements contributed to a reduction in KPMG's scores. Nevertheless, KPMG received average scores of 5.5 for Duke, 5.7 for SSM and 6.3 for Armstrong, compared with the score of 7 received by Deloitte for Suntax. There is only a gap of 1.5 to .7 points between Deloitte and KPMG's scores for SAP implementations, despite the deficient information within KPMG's corporate references. Key Staff Criterion: The proposals contain a summary of the experience of key staff and attached résumés. KPMG's proposed key staff person for Testing Lead was Frank Traglia. Mr. Traglia's summary showed that he had 25-years' experience respectively, in the areas of child support enforcement, information technology, project management and testing. Strange and Focht admitted that Traglia's résumé did not specifically list any testing experience. Mr. Focht further admitted that it was not unreasonable for evaluators to give the Testing Lead a lower score due to the lack of specific testing information in Traglia's résumé. Mr. Strange explained that the résumé was from a database of résumés. The summary sheet, however, was prepared by those KPMG employees who prepared the proposal. All of the evaluators resolved the conflicting information between the summary sheet and the résumé by crediting the résumé as more accurate. Each evaluator thought that the résumé was more specific and expected to see specific information regarding testing experience on the résumé for someone proposed as the Testing Lead person. Evaluators Addy and Ellis gave scores to the Testing Lead criterion of 4 and 5. Mr. Ron Vandenberg (evaluator 8) gave the Testing Lead a score of 9. Mr. Vandenberg was the only evaluator to give the Testing Lead a high score. The other evaluators gave the Testing Lead an average score of 4.2. The Vandenberg score thus appears anomalous. All of the evaluators gave the Testing Lead a lower score as it did not specifically list testing experience. Dr. Addy found that the summary sheet listed 25-years of experience in child support enforcement, information technology, and project management and system testing. As he did not believe this person had 100 years of experience, he assumed those experience categories ran concurrently. A strong candidate for Testing Lead should demonstrate a combination of testing experience, education and certification, according to Dr. Addy. Mr. Doolittle also expected to see testing experience mentioned in the résumé. When evaluating the Testing Lead, Mr. Bankirer first looked at the team skills matrix and found it interesting that testing was not one of the categories of skills listed for the Testing Lead. He then looked at the summary sheet and résumé from Mr. Traglia. He gave a lower score to Traglia as he thought that KPMG should have put forward someone with demonstrable testing experience. The evaluators gave a composite score to key staff based on the criteria in Table 8.2. In order to derive the composite score that he gave each staff person, Mr. Esser created a scoring system wherein he awarded points for each attribute in Table 8.2 and then added the points together to arrive at a composite score. Among the criteria he rated, Mr. Esser awarded points for CSE experience. Mr. Focht and Mr. Strange contended that since the term CSE experience is not actually listed in Table 8.2 that Mr. Esser was incorrect in awarding points for CSE experience in his evaluation. Table 8.2 does refer to relevant experience. There is no specific definition provided in Table 8.2 for relevant experience. Mr. Focht stated that relevant experience is limited to COTS/ERP experience, system development, life cycle and project management methodologies. However, these factors are also not listed in Table 8.2. Mr. Strange limited relevance to experience in the specific role for which the key staff person was proposed. This is a limitation that also is not imposed by Table 8.2. CSE experience is no more or less relevant than the factors posited by KPMG as relevant experience. Moreover, KPMG included a column in its own descriptive table of key staffs for CSE experience. KPMG must have seen this information as relevant if it included it in its proposal as well. Inclusion of this information in its proposal demonstrated that KPMG must have believed CSE experience was relevant at the time its submitted its proposal. Mr. Strange held the view that, in the bidders conference in a reply to a vendor question, the Department representative stated that CSE experience was not required. Therefore, Mr. Esser could not use such experience to evaluate key staff. Question 47 of the Vendor Questions and Answers, Volume 2 stated: QUESTION: In scoring the Past Corporate Experience section, Child Support experience is not mentioned as a criterion. Would the State be willing to modify the criteria to include at least three Child Support implementations as a requirement? ANSWER: No. However, a child support implementation that also meets the other characteristics (contract value greater than $5 million, serves a large number of users, includes data conversion from a legacy system and includes training development) would be considered "similar to CAMS CE." The Department's statement involved the scoring of corporate experience not key staff. It was inapplicable to Mr. Esser's scoring system. Mr. Esser gave the Training Lead a score of 1. According to Esser, the Training Lead did not have a ten-year résumé, for which he deducted one point. The Training Lead had no specialty certification or extensive experience and had no child support experience and received no points. Mr. Esser added one point for the minimum of four years of specific experience and one point for the relevance of his education. Mr. Esser gave the Project Manager a score of 5. The Project Manager had a ten-year résumé and required references and received a point for each. He gave two points for exceeding the minimum required informational technology experience. The Project Manager had twelve years of project management experience for a score of one point, but lacked certification, a relevant education and child support enforcement experience for which he was accorded no points. Mr. Esser gave the Project Liaison person a score of According to Mr. Focht, the Project Liaison should have received a higher score since she has a professional history of having worked for the state technology office. Mr. Esser, however, stated that she did not have four years of specific experience and did not have extensive experience in the field, although she had a relevant education. Mr. Esser gave the Software Lead person a score of 4. The Software Lead, according to Mr. Focht, had a long set of experiences with implementing SAP solutions for a wide variety of different clients and should have received a higher score. Mr. Esser gave a point each for having a ten-year résumé, four years of specific experience in software, extensive experience in this area and relevant education. According to Mr. Focht the Database Lead had experience with database pools including the Florida Retirement System and should have received more points. Mr. Strange concurred with Mr. Focht in stating that Esser had given low scores to key staff and stated that the staff had good experience, which should have generated more points. Mr. Strange believed that Mr. Esser's scoring was inconsistent but provided no basis for that conclusion. Other evaluators also gave key staff positions scores of less than 7. Dr. Addy gave the Software Lead person a score of 5. The Software Lead had 16 years of experience and SAP development experience as positive factors but had no development lead experience. He had a Bachelor of Science and a Master of Science in Mechanical Engineering and a Master's in Business Administration, which were not good matches in education for the role of a Software Lead person. Dr. Addy gave the Training Lead person a score of 5. The Training Lead had six years of consulting experience, a background in SAP consulting and some training experience but did not have certification or education in training. His educational background also was electrical engineering, which is not a strong background for a training person. Dr. Addy gave the subcontractor managers a score of 5. Two of the subcontractors did not list managers at all, which detracted from the score. Mr. Doolittle gave the Training Lead person a He believed that based on his experience and training it was an average response. Table 8.2 contained an item in which a proposer could have points detracted from a score if the key staff person's references were not excellent. The Department did not check references at this stage in the evaluation process. As a result, the evaluators simply did not consider that item when scoring. No proposer's score was adversely affected thereby. KPMG contends that checking references would have given the evaluators greater insight into the work done by those individuals and their relevance and capabilities in the project team. Mr. Focht admitted, however, that any claimed effect on KPMG's score is conjectural. Mr. Strange stated that without reference checks information in the proposals could not be validated but he provided no basis for his opinion that reference checking was necessary at this preliminary stage of the evaluation process. Dr. Addy stated that the process called for checking references during the timeframe of oral presentations. They did not expect the references to change any scores at this point in the process. KPMG asserted that references should be checked to ascertain the veracity of the information in the proposals. However, even if the information in some other proposal was inaccurate it would not change the outcome for KPMG. KPMG would still not have the required number of points to advance to the next evaluation tier. Divergency in Scores The Source Selection Plan established a process for resolving divergent scores. Any item receiving scores with a range of 5 or more was determined to be divergent. The plan provided that the Coordinator identify divergent scores and then report to the evaluators that there were divergent scores for that item. The Coordinator was precluded from telling the evaluator, if his score was the divergent score, i.e., the highest or lowest score. Evaluators would then review that item, but were not required to change their scores. The purpose of the divergent score process was to have evaluators review their scores to see if there were any misperceptions or errors that skewed the scores. The team wished to avoid having any influence on the evaluators' scores. Mr. Strange testified that the Department did not follow the divergent score process in the Source Selection Plan as the coordinator did not tell the evaluators why the scores were divergent. Mr. Strange stated that the evaluator should have been informed which scores were divergent. The Source Selection Plan merely instructed the coordinator to inform the evaluators of the reason why the scores were divergent. Inherently scores were divergent, if there was a five-point score spread. The reason for the divergence was self- explanatory. The evaluators stated that they scored the proposals, submitted the scores and each received an e-mail from Debbie Stephens informing him that there were divergent scores and that they should consider re-scoring. None of the evaluators ultimately changed their scores. Mr. Esser's scores were the lowest of the divergent scores but he did not re-score his proposals as he had spent a great deal of time on the initial scoring and felt his scores to be valid. Neither witnesses Focht or Strange for KPMG provided more than speculation regarding the effect of the divergent scores on KPMG's ultimate score and any role the divergent scoring process may have had in KPMG not attaining the 150 point passage score. Deloitte - Suntax Reference: Susan Wilson, a Child Support Enforcement employee connected with the CAMS project signed a reference for Deloitte Consulting regarding the Suntax System. Mr. Focht was concerned that the evaluators were influenced by her signature on the reference form. Mr. Strange further stated that having someone who is heavily involved in the project sign a reference did not appear to be fair. He was not able to state any positive or negative effect on KPMG by Wilson's reference for Deloitte, however. Evaluator Esser has met Susan Wilson but has had no significant professional interaction with her. He could not recall anything that he knew about Ms. Wilson that would favorably influence him in scoring the Deloitte reference. Dr. Addy also was not influenced by Wilson. Mr. Doolittle has only worked with Wilson for a very short time and did not know her well. He has also evaluated other proposals where department employees were a reference and was not influenced by that either. Mr. Ellis has only known Wilson from two to four months. Her signature on the reference form did not influence him either positively or negatively. Mr. Bankirer had not known Wilson for a long time when he evaluated the Suntax reference. He took the reference at face value and was not influenced by Wilson's signature. It is not unusual for someone within an organization to create a reference for a company who is competing for work to be done for the organization.
Recommendation Having considered the foregoing Findings of Fact, Conclusions of Law, the evidence of record and the pleadings and arguments of the parties, it is, therefore, RECOMMENDED that a final order be entered by the State of Florida Department of Revenue upholding the proposed agency action which disqualified KPMG from further participation in the evaluation process regarding the subject CAMS CE Invitation to Negotiate. DONE AND ENTERED this 26th day of September, 2002, in Tallahassee, Leon County, Florida. P. MICHAEL RUFF Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 SUNCOM 278-9675 Fax Filing (850) 921-6847 www.doah.state.fl.us Filed with Clerk of the Division of Administrative Hearings this 26th day of September, 2002. COPIES FURNISHED: Cindy Horne, Esquire Earl Black, Esquire Department of Revenue Post Office Box 6668 Tallahassee, Florida 32399-0100 Robert S. Cohen, Esquire D. Andrew Byrne, Esquire Cooper, Byrne, Blue & Schwartz, LLC 1358 Thomaswood Drive Tallahassee, Florida 32308 Seann M. Frazier, Esquire Greenburg, Traurig, P.A. 101 East College Avenue Tallahassee, Florida 32302 Bruce Hoffmann, General Counsel Department of Revenue 204 Carlton Building Tallahassee, Florida 32399-0100 James Zingale, Executive Director Department of Revenue 104 Carlton Building Tallahassee, Florida 32399-0100
The Issue The issues are whether Respondent is guilty of committing gross immorality or moral turpitude, in violation of Section 231.2615(1)(c), Florida Statutes; violating the Principles of Professional Conduct for the Education Profession, in violation of Section 231.2615(1)(i), Florida Statutes; or failing to maintain honesty in all professional dealings, in violation of Rule 6B-1.006(5)(a), Florida Administrative Code. If so, an additional issue is what penalty should be imposed.
Findings Of Fact At all material times, Respondent has held Florida Educator's Certificate 615429. Respondent is certified in business, drivers' education, and physical education. The School District of Palm Beach County hired Respondent to teach high-school business at Jupiter High School for the 1995-96 or 1996-97 school year. After changing schools with another teacher, the assistant principal of Respondent's new high school, Palm Beach Lakes High School, assigned Respondent to teach mathematics. Respondent has a very limited background in mathematics. Although he objected that he was not qualified to teach mathematics, he had no option but to accept the new assignment, or terminate his employment. Respondent reluctantly agreed to teach mathematics starting in the 1998-99 school year, but he was justifiably concerned about his ability to meet the needs of his mathematics students. In January 1999, Respondent walked past an unsecured room and saw a large number of test booklets in boxes stacked on a table in the school library. Respondent entered the room, picked up and examined a test booklet, and made a copy of the booklet before returning it to the table. The test booklet was the High School Competency Test (HSCT) that was being administered that year. Respondent claims to have copied the test booklet innocently, unaware that the test questions were not to be disclosed, except as was necessary to administer the test. Respondent also claims that he took the booklet to learn what generally he was supposed to be teaching and that he did not know that a future HSCT would be identical to the one that he had copied. Respondent's claims that he did not know that the test booklet was not to be removed or copied and that he took the booklet merely to learn what he was supposed to teach in general are discredited as highly unlikely. If Respondent had thought that the test booklets were freely available to teachers, he would have merely taken one, not copied one and returned it to the table. Respondent never asked for a booklet, nor did he ever disclose to anyone else at the school that he had taken a copy of a booklet. From the start, Respondent knew that his possession of the test booklet was improper. Respondent's claim that he did not know anything about the HSCT, such as its importance or confidentiality, undermines his claim that he took a copy of the test booklet to learn what to teach in mathematics. At the time, students had to pass the HSCT to graduate from high school. Respondent likely knew this fact, otherwise, he would not have relied so heavily upon this test booklet as the source of information as to what he had to teach in mathematics. Rather than taking his cue as to what to teach from the mathematics textbook or from other mathematics teachers, Respondent took the shortcut of obtaining the ultimate test instrument and relying on the test contents for deciding what to teach in his mathematics class. On the other hand, Respondent did not know that the identical test would be administered again. This fact was not widely known by teachers or even administrators. Once he had examined the test booklet, Respondent worked out the answers, although he required assistance to do so. He then cut and pasted questions onto worksheets for use by his students, who would complete the worksheets in class and turn them into Respondent, who would go over the answers in class. The investigator of The School District of Palm Beach County concludes that Respondent's rearranging of questions is part of his attempt to conceal his wrongdoing. This conclusion is incorrect, as the rearranging of questions allowed Respondent to save copying costs. The evidence likewise fails to establish that Respondent told his students not to disclose the worksheets. Thus, the sole evidence of concealment is Respondent's failure to disclose his possession of the HSCT booklet to administrators or other teachers. In fact, once confronted with his possession of the HSCT, Respondent admitted to his wrongdoing and cooperated with the investigation. However, it is impossible to harmonize Respondent's claims of innocence and good faith with the proximity of his use of the copied test with the test date. If, as Respondent claims, he intended only to learn what he should be teaching in mathematics, he could have examined the copied test booklet, noted the areas covered, and covered them in an orderly fashion through the school year, using different questions from those found in his copy of the test booklet. Instead, Respondent gave his students numerous questions from his copy of the test booklet on September 24 and 26-29 and October 1. The presentation of a variety of mathematical concepts in such close proximity to the HSCT test date suggest a knowing misuse of the copied test booklet. Respondent's knowing misuse of the test, combined with the chance occurrence of the administration of the same test in October 2000, led to distorted results among his students, many of whom recognized that questions on the real test were identical with questions with which Respondent had prepared them. After an investigation, the Florida Department of Education and The School District of Palm Beach County decided to invalidate the mathematics scores of the hundreds of students at Respondent's high school who had taken the October 2000 HSCT and require them to retake a different version of the mathematical portion of the test. The question naturally arises whether October 2000 marked the first time that Respondent used the HSCT booklet that he had taken in January 1999. Respondent claims that he filed the test booklet and forgot about it until shortly before the October 2000 test. The investigation revealed that the scores of Respondent's students on the mathematics portion of the HSCT during the 1999-2000 school year were considerably better than the scores of similarly situated students, but investigators lacked the evidence to pursue this matter further. Thus, the evidence fails to establish that Respondent improperly used the copied test material more than once. Petitioner's reliance on Respondent's training as a proctor does not tend to establish Respondent's knowledge of his misuse of the test booklet that he copied. The training materials do not directly address older testing materials in the possession of a proctor, and Respondent possesses only limited ability to draw the inferences that Petitioner claims were inescapable. Also, the late recollection of one of Petitioner's witnesses that Respondent had inquired whether he might obtain a bonus if his students performed well on the HSCT is discredited. Petitioner has proved that Respondent obtained a copy of an HSCT under circumstances that he knew were improper, and he knowingly misused the copied test materials to prepare his students to take the HSCT. Undoubtedly, Respondent did not know that the October 2000 HSCT would be identical to the test that he had copied. Also, Petitioner has failed to prove that Respondent tried to conceal his misuse of the copied HSCT materials, other than by not mentioning to an administrator or other teacher that he possessed these materials. Lastly, Petitioner has failed to prove that Respondent's actions were motivated by self-interest. Respondent doubted his ability to teach mathematics, and he misused the test materials to serve the interests of his students, although at the expense of thousands of other students whose preparation did not include exposure to HSCT prior to taking it. Undoubtedly, this commitment to his students is partly responsible for the testimony of Respondent's principal, who described him as an "outstanding teacher," although Respondent received a decidedly mixed review from the four students whom he called as witnesses on his behalf. After an investigation, the Superintendent of The School District of Palm Beach County recommended to the School Board that it suspend Respondent without pay for ten days. The School Board adopted this recommendation. This is the only discipline that Respondent has received as a teacher, and he proctored last school year the Florida Comprehensive Assessment Test, which has replaced the HSCT.
Recommendation It is RECOMMENDED that the Education Practices Commission enter a final order finding Respondent guilty of failing to maintain honesty in all professional dealings, in violation of Rule 6B-1.006(5)(a), Florida Administrative Code; suspending his Educator's Certificate for six months; and placing his certificate on probation for three years. DONE AND ENTERED this 19th day of September, 2003, in Tallahassee, Leon County, Florida. S ROBERT E. MEALE Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 SUNCOM 278-9675 Fax Filing (850) 921-6847 www.doah.state.fl.us Filed with the Clerk of the Division of Administrative Hearings this 19th day of September, 2003. COPIES FURNISHED: Kathleen M. Richards, Executive Director Florida Education Center Department of Education 325 West Gaines Street, Room 224-E Tallahassee, Florida 32399-0400 Daniel J. Woodring, General Counsel Department of Education 325 West Gaines Street 1244 Turlington Building Tallahassee, Florida 32399-0400 Marian Lambeth, Program Director Bureau of Educator Standards Department of Education 325 West Gaines Street, Suite 224-E Tallahassee, Florida 32399-0400 Charles T. Whitelock Whitelock & Associates, P.A. 300 Southeast 13th Street Fort Lauderdale, Florida 33316 Matthew E. Haynes Chambleee, Johnson & Haynes, P.A. The Barrister's Building, Suite 500 1615 Forum Place West Palm Beach, Florida 33401
The Issue Petitioner has challenged her grade on the written portion of the September 1990 Optometry licensing examination. The issue for determination is whether she is entitled to a passing grade. BACKGROUND, FINDINGS OF FACT AND RECOMMENDED DISPOSITION The hearing was convened as scheduled, and the parties proceeded with their argument and presentation of exhibits and testimony. Respondent stipulated that Ms. Woods had passed the clinical and practical portions of the examination, but received a 68.5% score on the written portion of the examination. A passing score is 70%. Ms. Woods required three additional raw score points to pass. During the course of the hearing it became apparent that the text of one of the questions challenged by Ms. Woods was misleading, as the correct answer in a multiple choice series was misspelled. The misspelling was such that the proper spelling could have been either the term intended by the test, or another term that would have been an incorrect answer. Ms. Woods selected the next best answer in the series. After a brief recess in the hearing, Respondent stipulated that Petitioner should be given credit for her answer on that question, as well as ensuing questions that were part of the same hypothetical example. As stipulated by Respondent on the record, this results in a passing score for Petitioner. It was agreed that a Recommended Order would be entered, consistent with this stipulation, and that the examination questions received in evidence would be forwarded to the Board, appropriately sealed.
Recommendation Based on the foregoing, it is hereby, recommended that the Board of Optometry enter its final order granting a passing score on the September 1990 Optometry examination to Petitioner, Bernadette Susan Woods. RECOMMENDED this 19th day of July, 1991, in Tallahassee, Leon County, Florida. MARY CLARK Hearing Officer Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-1550 (904)488-9675 Filed with the Clerk of the Division of Administrative Hearings this 19th day of July, 1991. COPIES FURNISHED: Bernadette S. Woods 315 Lakepointe Drive, #104 Altamonte Springs, FL 32701 Vytas J. Urba, Asst. General Counsel Department of Professional Regulation 1940 N. Monroe Street Tallahassee, FL 32399-0792 Patricia Guilford, Executive Director Board of Optometry Department of Professional Regulation 1940 N. Monroe Street Tallahassee, FL 32399-0792 Jack McRay, General Counsel Department of Professional Regulation 1940 N. Monroe Street Tallahassee, FL 32399-0792
The Issue Whether Ketura Bouie suffers from “retardation”, as that term is defined by Section 393.063(43), Florida Statutes, and therefore qualifies for developmental services offered by the Respondent agency under Chapter 393, Florida Statutes.
Findings Of Fact Ketura Bouie is 15 years old. She currently resides in Tallahassee, Florida. She is enrolled in a new school after transferring from Chatahoochee. Ketura has had several “social” promotions from grade to grade over the years. Her application for developmental services has been denied by the Respondent agency. Wallace Kennedy, Ph.D., is a Board-certified and Florida-licensed clinical psychologist. He was accepted as an expert in clinical psychology and the testing of children. He conducted a psychological evaluation of Ketura on April 12, 1995, for which he has provided a written narrative dated April 13, 1995. His narrative was admitted in evidence. Ketura was 13 years old at the time of Dr. Kennedy’s evaluation. He administered three standardized tests which are recognized and accepted for determining applicants’ eligibility for developmental services. These tests were: a wide range achievement test, Wechsler Intelligence Scale for Children— Revised (WISC-R), and Vineland Adaptive Behavior Scale. (Vineland) The wide range achievement test generally measures literacy. Ketura recognized only half of the upper-case letters of the alphabet and only a few three-letter kindergarten words. Her results indicated that she has the achievement level expected of a five and a half year old kindergarten student, even though she was then placed in the seventh grade. In Dr. Kennedy's view, there is "no chance Ketura will become functionally literate". The WISC-R measures intellectual functioning and academic aptitude without penalizing the child for handicaps. The mean score on this test is 100. To score two or more deviations from this mean, a subject must score 70 or below. All of Ketura’s WISC-R scores on the test administered by Dr. Kennedy in April 1995 were well below 70. They consisted of a verbal score of 46, a performance score of 46, and a full scale score of 40. Ketura’s full scale IQ of 40 is in the lowest tenth of the first percentile and represents a low moderate level of mental retardation. Ketura’s full scale score of 40 is the lowest result that WISC-R can measure. The Vineland measures communication, daily living skills, and socialization. Ketura’s composite score for Dr. Kennedy on the Vineland was 42. In conducting the Vineland test, Dr. Kennedy relied on information obtained through his own observation of Ketura and information obtained from Ketura’s mother. It is typical in the field of clinical psychology to rely on information supplied by parents and caregivers, provided they are determined to be reliable observers. Dr. Kennedy assessed Ketura’s mother to be a reliable observer. Dr. Kennedy’s Vineland test revealed that Ketura has a social maturity level of about six years of age. Her verbal and written communication skills are poor. Ketura has poor judgment regarding her personal safety. She cannot consistently remember to use a seatbelt and cannot safely use a knife. She has poor domestic skills. She has no concept of money or of dates. She does not help with the laundry or any other household task. She cannot use the phone. Ketura’s socialization skills are also poor. She does not have basic social manners. Her table manners and social interactive skills are poor. She has no friends, and at the time of Dr. Kennedy’s evaluation, she was unhappy due to classmates making fun of her for being unable to recite the alphabet. Dr. Kennedy rendered an ultimate diagnosis of moderate mental retardation and opined that Ketura's retardation is permanent. Although Dr. Kennedy observed that Ketura was experiencing low levels of depression and anxiety during his April 1995 tests and interview, he did not make a clinical psychological diagnosis to that effect. He attributed these emotional components to Ketura’s lack of confidence in being able to perform the tasks required during testing. In his opinion, Ketura did not have any behavioral or emotional problems which interfered with the reliability of the tests he administered. Also, there were no other conditions surrounding his evaluation which interfered with the validity or reliability of the test scores, his evaluation, or his determination that Ketura suffers from a degree of retardation which would qualify her for developmental services. In Dr. Kennedy’s expert opinion, even if all of Ketura's depression and anxiety were eliminated during testing, her WISC-R scores would not have placed her above the retarded range in April 1995. The retardation range for qualifying for developmental services is 68 or below. Ketura’s I.Q. was tested several times between 1990 and April 1995 with resulting full scale scores ranging from 40 to All or some of these tests and/or reports on the 1990 - 1995 tests were submitted to the agency with Ketura’s application for developmental services. Also included with Ketura’s application to the agency were mental health reports documenting depression, a recognized mental disorder. The most recent of these was one done as recently as May of 1996. However, none of these reports were offered or admitted in evidence at formal hearing. Respondent’s sole witness and agency representative, was Ms. JoAnne Braun. She is an agency Human Service Counselor III. Ms. Braun is not a Florida-licensed psychologist and she was not tendered as an expert witness in any field. As part of the application process, she visited with Ketura and her mother in their home. She also reviewed Petitioner’s application and mental health records described above. She reviewed the fluctuating psychological test scores beginning in 1990, one of which placed Ketura at 70 and another of which placed her at 74 on a scale of 100. Ms. Braun also reviewed a March 1995 psychological testing series that showed Ketura had a verbal 50, performance 60, and full scale 62 on the WISC-R test, one month before Dr. Kennedy’s April 1995 evaluation described above. However, none of these items which she reviewed was offered or admitted in evidence. The agency has guidelines for assessing eligibility for developmental services. The guidelines were not offered or admitted in evidence. Ms. Braun interpreted the agency's guidelines as requiring her to eliminate the mental health aspect if she felt it could depress Ketura's standard test scores. Because Ms. Braun "could not be sure that the mental health situation did not depress her scores," and because the fluctuation of Ketura’s test scores over the years caused Ms. Braun to think that Ketura’s retardation might not “reasonably be expected to continue indefinitely”, as required by the controlling statute, she opined that Ketura was not eligible for developmental services. Dr. Kennedy's assessment and expert psychological opinion was that if Ketura's scores were once higher and she now tests with lower scores, it might be the result of better testing today; it might be due to what had been required and observed of her during prior school testing situations; it might even be because she was in a particularly good mood on the one day she scored 70 or 74, but his current testing clearly shows she will never again do significantly better on standard tests than she did in April 1995. In his education, training, and experience, it is usual for test scores to deteriorate due to a retarded person's difficulties in learning as that person matures. I do not consider Ms. Braun’s opinion, though in evidence, as sufficient to rebut the expert opinion of Dr. Kennedy. This is particularly so since the items she relied upon are not in evidence and are not the sort of hearsay which may be relied upon for making findings of fact pursuant to Section 120.58(1)(a), Florida Statutes. See, Bellsouth Advertising & Publishing Corp. v. Unemployment Appeals Commission and Robert Stack, 654 So.2d 292 (Fla. 5th DCA 1995); and Tenbroeck v. Castor, 640 So.2d 164, (Fla. 1st DCA 1994). Particularly, there is no evidence that the "guidelines" (also not in evidence) she relied upon have any statutory or rule basis. Therefore, the only test scores and psychological evaluation upon which the undersigned can rely in this de novo proceeding are those of Dr. Kennedy. However, I do accept as binding on the agency Ms. Braun’s credible testimony that the agency does not find that the presence of a mental disorder in and of itself precludes an applicant, such as Ketura, from qualifying to receive developmental services; that Ketura is qualified to receive agency services under another program for alcohol, drug, and mental health problems which Ketura also may have; and that Ketura’s eligibility under that program and under the developmental services program, if she qualifies for both, are not mutually exclusive.
Recommendation Upon the foregoing findings of fact and conclusions of law, it is RECOMMENDED that the Department of Children and Families issue a Final Order awarding Ketura Bouie appropriate developmental services for so long as she qualifies under the statute.RECOMMENDED this 24th day of February, 1997, at Tallahassee, Florida. ELLA JANE P. DAVIS Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-1550 (904) 488-9675 SUNCOM 278-9675 Fax FILING (904) 921-6847 Filed with the Clerk of the Division of Administrative Hearings this 24th day of February, 1997. COPIES FURNISHED: Gregory D. Venz, Agency Clerk Department of Children and Families Building 2, Room 204 1317 Winewood Blvd. Tallahassee, FL 32399-0700 Richard A. Doran General Counsel Building 2, Room 204 1317 Winewood Blvd. Tallahassee, FL 32399-0700 Marla Ruth Butler Qualified Representative Children's Advocacy Center Florida State University Tallahassee, FL 32302-0287 Marian Alves, Esquire Department of Health and Rehabilitative Services 2639 North Monroe Street Suite 100A Tallahassee, FL 32399-2949
The Issue Is Petitioner entitled to receive developmental services from the Department of Children and Family Services (the Department), due to his developmental disability based on retardation, pursuant to Chapter 393, Florida Statutes.
Findings Of Fact Petitioner was born on June 21, 1979, and at the time of the hearing was 21 years of age. Petitioner was evaluated at the University Hospital of Jacksonville in Jacksonville, Florida, in 1982, at two and one- half years of age. A report from that evaluation indicated that Petitioner was afflicted with a seizure disorder, speech delay, and right-sided dysfunction. During September, 1985, at age six years, three months, he was evaluated at the Hope Haven Children's Clinic, in Jacksonville, Florida. His hearing was tested and determined to be normal. A psychological evaluation noted that his communication skills were below his age level. A report of this evaluation indicates he was a slow learner with weaknesses in processing, retaining, and retrieving information, particularly in the area of speech and language development. On January 21, 1986, Petitioner was again evaluated at the Hope Haven Children's Clinic in Jacksonville, Florida. It was noted at that time he had difficulty in following directions and performing in a regular school environment. He was far behind his classmates academically. During this evaluation he was administered a Peabody Individual Achievement Test and received a standard score of 75 on both mathematics and reading recognition. These scores are above the range of retardation. Petitioner was examined by the School Psychology Services Unit, Student Services, of the Duval County School Board, on February, 17, 1987, when he was seven years and seven months of age. At the time he was receiving "specific learning disabilities full time services" while at Englewood Elementary School in Jacksonville, Florida. It was noted that he was difficult to evaluate because he was easily distracted. During the evaluation, on the Wechsler Intelligence Scale for Children-Revised, Petitioner received a full-scale intelligence quotient (IQ) of 74. This score indicated that he was below average within his verbal abstract reasoning, verbal expression and practical knowledge, visual attentiveness, visual analysis and visual synthesis. He was determined to be within the "slow learner's" range of development. Petitioner was referred to School Social Work Services, Duval County School Board in Jacksonville, Florida, on January 9, 1990. He was referred to the School Psychology Services Unit, where a Wechsler Intelligence Scale for Children-Revised was administered on June 11, 1990. His full-scale IQ was determined to be 74. He was also administered a Vineland assessment, which measures adaptive behavior rather than intelligence. In this case, Petitioner's mother provided answers regarding Petitioner's behavior and adaptability. At the time of this assessment, he was almost 11 years of age. On December 18, 1996, Petitioner was evaluated by the Sand Hills Psychoeducational Program in Augusta, Georgia, when he was 17 years of age. He was administered a battery of tests. The WAIS-R indicated borderline intellectual ability, but not retardation. The Stanford-Binet was 56. This score included a verbal reasoning score of 58 and an abstract visual reasoning score of 72. The split in the scores generates doubt as to the validity of the test. Psychologist Cydney Yerushalmi, Ph.D., an expert witness for the Department, and psychologist Barbara Karylowski, Ph.D., an expert witness for the Petitioner, opined that the Stanford Binet was inappropriate for a person who had attained the age of 17 because it would tend to produce lower scores. Dr. Karylowski tested Petitioner's IQ in February and March 2000. She concluded that Petitioner had a full-scale IQ of 68, which is mild retardation. At the time of that test Petitioner was 20 years of age. Dr. Karylowski opined that the scores she obtained were consistent with all of the scores she had obtained in prior testing because the confidence interval for his IQ was 68 to 77. This would place Petitioner within the range of retardation. Two standard deviations from the mean IQ is 70. It is Dr. Karylowski's opinion that Petitioner is mildly retarded. Her opinion is based on criteria set forth in the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV), published by the American Psychiatric Association. The DSM-IV definition of retardation requires significantly subaverage general intellectual functioning that is accompanied by significant limitations in adaptive behavior. The onset must occur before the age of 18 years. Accordingly, she believes that Petitioner was retarded before attaining the age of 18. Luis Quinones, M.D., was accepted as an expert witness in the field of psychiatry. Dr. Quinones opined that Petitioner meets the definition of mental retardation under DSM-IV. This means that he believes Petitioner was retarded before attaining the age of 18. He gave great weight to the Petitioner's lack of adaptive functioning in forming this opinion. Dr. Quinones opined that the definition of retardation in the DSM-IV is essentially the same as that found in Section 393.063(44), Florida Statutes. Dr. Yerushalmi evaluated Petitioner and reviewed his medical records. She administered the Wechsler Adult Intelligence Scale, Third Edition, to Petitioner, on August 11, 1999. Petitioner had a verbal score of 74, a performance score of 75, and a full-scale score of 72. She opined that Petitioner was not retarded under the definition set forth in Section 393.063(44), Florida Statutes. The aforementioned statute requires that one must meet a threshold of two or more standard deviations from the mean IQ, in order to be classified as retarded. All measures of IQ have a statistical confidence interval or margin of error of approximately five points, according to the DSM-IV. If one accepts the lower range of the confidence interval of the scores Petitioner has attained over the years, then he meets the two or more standard deviation threshold. Acceptance of the upper limits of the confidence interval would indicate that Petitioner clearly does not fall within the range of retardation. The significance of the confidence interval is reduced substantially when test results produced over a long period of time, by different test administrators, all indicate that Petitioner's IQ is not two or more standard deviations from the mean. Petitioner was diagnosed with many disorders by a variety of practitioners prior to the age of 18. No diagnosis of mental retardation was ever made. At age 21, Petitioner often behaves as if he were much younger, has focused on an 11-year-old as a girlfriend, and may become violent when not properly medicated. He likes to play pretend games of the sort that one would think would entertain a child. For instance, he likes to pretend that he is a law enforcement officer when he rides in a car. He prefers interacting with children who are five or six years younger. He is deficient in the area of personal hygiene. He likes to act silly. He is incapable of holding a driver's license. At the time of the hearing, Petitioner lived with his aunt, Ms. Mary Margaret Haeberle, who is a special education school teacher. She has provided a nurturing environment for Petitioner. Although Petitioner's parents divorced when he was a child, they have worked to address his needs. His younger sister understood Petitioner's problems and attempted to ameliorate them. Upon consideration all of the evidence, it is found that Petitioner was not possessed of an IQ which was two or more standard deviations from the mean. Therefore, there is no need to consider his adaptive function in concluding that he is not retarded.
Recommendation Based upon the foregoing Findings of Fact and Conclusions of Law set forth herein, it is RECOMMENDED: That the Department enter a final order finding that Petitioner is not entitled to receive developmental services due to a developmental disability based on retardation. DONE AND ENTERED this 4th day of January, 2001, in Tallahassee, Leon County, Florida. HARRY L. HOOPER Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 SUNCOM 278-9675 Fax Filing (850) 921-6847 www.doah.state.fl.us Filed with the Clerk of the Division of Administrative Hearings this 4th day of January, 2001. COPIES FURNISHED: Michael R. Yokan, Esquire 1301 Riverplace Boulevard, Suite 2600 Jacksonville, Florida 32207 Robin Whipple-Hunter, Esquire Department of Children and Family Services 5920 Arlington Expressway Jacksonville, Florida 32231-0083 Virginia A. Daire, Agency Clerk Department of Children and Family Services Building 2, Room 204B 1317 Winewood Boulevard Tallahassee, Florida 32399-0700 Josie Tomayo, General Counsel Department of Children and Family Services 1317 Winewood Boulevard Building 2, Room 204 Tallahassee, Florida 32399-0700
The Issue The issue in this case is whether Petitioner cheated or otherwise acted inappropriately during the Florida Teacher Certification Examination on July 25, 2009.
Findings Of Fact Ghobrial is a native of Sudan, who migrated to the United States in November 1993. Her native language is Arabic. Ghobrial has a bachelor's degree in philosophy, with a minor in child psychology from the University of Cairo, Egypt. Ghobrial, her husband, and all four of her children have advanced college degrees. Education is important to Ghobrial. Within months of her arrival in the United States, Ghobrial obtained employment as a pre-kindergarten teacher at Creative Learning Center in Winter Park, Florida. Inasmuch as Ghobrial was at that time still trying to master the English language, her primary responsibilities were caring for infants and toddlers. Ghobrial worked at the center for six years, during which time she became assistant director of the child care center. She then worked as a paraprofessional in the Orange County School System for four years. In 2006, Ghobrial was given the opportunity to teach first and second grades at Arbor Ridge Elementary School in Orlando, Florida, under a temporary teaching certificate. That certificate had expired as of the date of the final hearing in this matter. Beginning in March 2003, Ghobrial began her efforts to pass the Exam so that she could obtain a permanent teaching license. The paper-based version of the Exam is offered several times throughout the year in different locations. The Exam consists of four subtests, one of which involves the writing of an original essay on one of two possible topics. The purpose of the essay portion of the Exam is to "demonstrate an examinee's ability, in the time allotted, to compose and write an original essay that completely addresses the topic in an effective, well-organized manner, with good grammar and spelling." Ghobrial quickly realized that the Exam was written in such a way that her unfamiliarity with the English language was a large impediment. It took several tries before she began to obtain passing grades for any portions of the Exam. Try as she may, however, Ghobrial was unable to master the essay portion of the Exam. Ghobrial took the essay portion of the Exam 12 times prior to the test at issue. During the time she was taking the essay portion, she took the other portions as well. She passed the Prekindergarten/Primary 3K Examination on October 22, 2005. On January 20, 2007, she passed the Professional Education portion of the Exam. On January 12, 2008, Ghobrial passed the General Knowledge Subtest 4: Mathematics, portion of the Exam. The General Knowledge Subtest 2: English Language Skills was passed on October 25, 2008. Then, on April 4, 2009, Ghobrial passed the General Knowledge Subtest 3: Reading, portion of the Exam. All that remained was to pass the General Knowledge Subtest 1: Essay. Ghobrial took every step available to her to ensure ultimate passage of the essay portion. She practiced her writing using on-line test-taking sites. She enrolled in an English prep course at Valencia Community College ("Valencia") in 2008 and a college prep reading course in 2009. She also enrolled in a Freshman Comp I course at Valencia for additional education and training. Meanwhile, she obtained private tutoring as part of her unyielding efforts to pass the essay portion of the Exam. Each person taking the Exam is provided with registration materials which sets forth how the Exam will be administered and the purpose of the Exam and prescribes what actions or behaviors will be deemed improper. One portion of the registration materials (which Ghobrial admittedly received) defines the following activity as cheating: "During the examination administration, writing an essay that shows evidence of having been prepared before the examination; that is, presenting an essay that is not an original essay composed by the examinee during the test in direct and specific response to an essay topic presented on the test." The Essay portion of the Exam was set up to test the exam takers' ability to organize, write, spell, and correctly insert grammar into a composition of their own creation. Each time the test was given, there would be two essay topics from which to choose. The topics were general and open-ended. It was not the intent of the test to measure one's knowledge about a particular subject; rather, it was to determine how well the person could correctly put their thoughts and ideas on paper. Essay topics would be general topics from which each examinee could narrow the focus as much as he or she saw fit. It was important for the Exam graders to be consistent with their review of the essays. The graders had to calibrate their reviews so that each grader was looking for the same grammatical and structural content. In order to do that, the list of essay topics had to be fairly limited so that graders could be trained as to those particular essay questions. Having too many different topics would not allow for a uniform review of all essays. At the July 25, 2009, Exam, two essay topics were offered; one of them was "An invention in the field of science or technology that influences people's lives." Ghobrial had seen this topic several times in past exams, and it was even a topic she had seen during her private tutoring practice sessions. She had even practiced writing a similar essay in the recent past. Thus, the essay topic was very familiar to her and gave her some hope that this familiarity would result in a passing score.1 As she had done in her practice essays, Ghobrial narrowed the scope of the essay down by focusing on one particular invention--computers. When Ghobrial took the essay portion of the Exam on October 25, 2008, this same topic was offered. Ghobrial's essay in that exam concentrated on the television as an invention influencing people's lives. On the July 25, 2009, exam, Ghobrial wrote about computers as an invention that had influenced people's lives. When Ghobrial saw this topic during her exam preparation courses, she switched to computers as the influential invention. She wrote the essay at the Exam site without benefit of notes or other aids. She did, however, remember some of what she had written on practice essays and surely that may have influenced what she wrote in her Exam essay on the day in question. Ghobrial's essay was reviewed by two graders: One gave her a grade of three and the other gave her a grade of five. A combined grade of six is considered passing. However, any time there is a discrepancy of more than one number by two graders, another grader must review the essay as well. That being the case, Ghobrial's essay was selected for review by the chief grader assigned to that particular grading session. During the time that Ghobrial's essay was being graded by the chief grader, Respondent was made aware that there appeared to be some essays written by examinees that were too similar in content to be coincidence. That being the case, graders were cautioned to be on the lookout for any essay bearing similarity to the suspected copies. When the chief grader did her re-review of Ghobrial's essay, she found there to be similarities between Ghobrial's essay and two other essays. As a result Ghobrial was given a grade of Invalid for her essay. (The other two essays were also given a grade of Invalid, but neither of those examinees contested their grade.) Ghobrial's essay contained the following sentences or phrases that were deemed suspect because of their similarity to certain on-line essays (and presumably to the two other suspect essays, as well): "'Computer is an electronic brain' as compared to the on-line version of the essay which included the term 'computer is a wonderful electronic brain'." "'Has changed our lives through dramatic advances in science and medicine, business, and education' versus 'has significantly changed our world through advances in science and medicine, business and education'." "'Clearly, so many advances in science and medicine are due to computers in the last fifty years' versus 'in the last fifty years'," which Ghobrial had written in a practice exam. "'Clearly, no modern office could operate as efficiently without a variety of computer equipments [sic]' versus 'Indeed, offices and banks can't operated as efficiently, without a variety of computers'." "'Thus, both teachers and students have come to rely on computers in a way which was not possible 40 years ago' versus 'Thus, both students and teachers have come to rely on computers to enhance learning in a way not possible years ago'." "'We landed on the moon and returned safely all because of computers' versus 'landed men on the moon because of computers'." The structure of the three essays was also similar. Each contained the following: An introductory paragraph identifying computers as the topic of the essay; Paragraph 2 discussing science and medicine; Paragraph 3 referring to business; Paragraph 4 with a reference to education; and A summary paragraph with an almost identical phraseology. Those phrases and sentences, when compared to the organization of the on-line essay, caused the graders to take a longer look at Ghobrial's essay. They found too many similarities to be deemed a coincidence, thus drawing the conclusion that Ghobrial must have cheated. Of the 32 sentences in Ghobrial's essay, about 12 of them had some similarity to the on-line essay. (Respondent asserts there are 20 sentences with similarities, but that contention is not completely borne out by the facts when reviewing the essays subjectively.) Ghobrial admittedly used on-line practice exams to prepare for the Exam. There is nothing inherently wrong with doing that; it is absolutely important for a person to prepare for the Exam using all means within his or her grasp. One of the on-line services called "123helpme.com," included an essay entitled "Computers of the World." Some of the language in the on-line essay is strikingly similar to what Ghobrial wrote during the Exam. The two essays are not verbatim, nor is there complete similarity of content. But they are similar. Ghobrial does not deny that she may have seen the on-line essay. She was preparing for the Exam as best she could and would have relied on any available source. There is no evidence as to whether the other two suspect examinees saw the same on-line essay practice test, but that inference could be drawn from the similarity of their work. However, Ghobrial did not memorize the essay, nor is there any indication she had a copy of the essay to refer to while she was taking the Exam. Rather, Ghobrial had some recall of the topic based on the fact that she had studied it as part of her preparation. It is a very close call as to whether the essay written at the time of the July 25, 2009, Exam was "original" in all aspects. The essay was obviously influenced by Ghobrial's preparation for the Exam; but just as obviously, she drafted the essay without benefit of notes, and it was "original" in the form submitted for the Exam. That is, the essay was drafted during the Exam time under close supervision and monitoring. Ghobrial, when testifying on her own behalf, was very credible and sincere. She is still working to master the English language, but showed sufficient understanding to have crafted the essay as it was submitted. Respondent's witnesses were no less credible. Each of them expressed their opinions based on their training and experience. And although Respondent's witnesses established the basis for their determination that Ghobrial had "cheated" by memorizing the essay, there is no direct evidence to support their final contention.
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is RECOMMENDED that a final order be entered by Respondent, Dr. Eric J. Smith, as Commissioner of Education, deeming Petitioner, Ciciel Ghobrial's, essay to have been an original composition deserving of a passing grade. DONE AND ENTERED this 12th day of July, 2010, in Tallahassee, Leon County, Florida. R. BRUCE MCKIBBEN Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 Fax Filing (850) 921-6847 www.doah.state.fl.us Filed with the Clerk of the Division of Administrative Hearings this 12th day of July, 2010.
