The Issue Whether Petitioner is entitled to additional credit for his responses to the "Principles and Practice" portion of the electrical engineer examination administered by Respondent in October 1996.
Findings Of Fact Petitioner took the electrical engineer licensing examination administered by Respondent in October 1996. Respondent is the agency of the State of Florida with the duty to regulate the practice of electrical engineering in Florida. Pursuant to Section 471.015, Florida Statutes, an applicant for licensure as an electrical engineer is required to successfully pass both parts of a licensure examination.1 The electrical engineer licensure examination at issue in this proceeding was developed and graded by the National Council of Examiners for Engineering and Surveying (NCEES). Following the initial grading of the "Principles and Practice" section of the exam, Petitioner was awarded a score of 68. A total score of 70 was required to pass that portion of the examination. Petitioner thereafter timely challenged the grading of two questions on the "Principles and Practice" portion of the exam. His challenge was limited to Questions 130 and 132. Petitioner did not specifically challenge Question 131. In response to that challenge, Respondent sent Petitioner’s examination package back to NCEES to have the "Principles and Practice" portion of the examination re-graded. NCEES re-graded all of Petitioner's answers to the "Principles and Practice" portion of the examination, including his responses to Questions 130, 131, and 132. NCEES initially awarded Petitioner a score of 2 points for his answer to Question 130. When the answer was re-graded, Petitioner was not awarded any additional credit for his answer to Question 130. The record in this proceeding established that Petitioner's answer to Question 130 was properly re-graded. Petitioner is not entitled to any additional credit for his response to Question 130. NCEES initially awarded Petitioner a score of 2 points for his answer to Question 132. When the answer was re-graded, Petitioner was awarded a score of 4 points for his answer to Question 132. The record in this proceeding established that Petitioner's answer to Question 132 was properly re-graded. Petitioner is entitled to a score of 4 points for his answer to Question 132. NCEES initially awarded Petitioner a score of 8 points for his answer to Question 131. When the answer was re-graded, Petitioner was awarded a score of 6 points for his answer to Question 131. The record in this proceeding established that Petitioner's answer to Question 131 was properly re-graded. Petitioner is entitled to a score of 6 points for his answer to Question 131. Petitioner is not entitled to a score of 8 for his answer to Question 131. Each of the three questions at issue in this proceeding is a problem that requires multiple steps and computations to solve. If a candidate correctly answers all parts of the question a score of 10 points is awarded. Partial credit can be awarded based on how many of the parts of the question are correctly answered. There is no allegation that the three questions involved in this proceeding are ambiguous or otherwise inappropriate for a licensure examination. The record is not clear when Respondent notified Petitioner of its position following the re-grading of the questions at issue. It is clear that Petitioner was aware of Respondent's position prior to the start of the formal hearing. During the formal hearing and in his post-hearing submittal, Petitioner challenged Respondent's right to re-grade Question 131 since he had not specifically challenged that question. Petitioner has not asserted that he was provided insufficient notice of Respondent's position.
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is RECOMMENDED that Respondent enter a Final Order that awards Petitioner a score of 68 on the "Principles and Practice" portion of the October 1996 licensure examination. DONE AND ENTERED this 11th day of December, 1997, in Tallahassee, Leon County, Florida. CLAUDE B. ARRINGTON Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (904) 488-9675 SUNCOM 278-9675 Fax Filing (904) 921-6847 Filed with the Clerk of the Division of Administrative Hearings this 11th day of December, 1997.
The Issue The issues in this case are whether Petitioner, Gary P. Santoro (“Petitioner” or “Mr. Santoro”), undeservedly received a failed grade on the Construction Business and Finance Examination (“Examination”) for licensure as an air-conditioning contractor; whether any questions on the examination had more than one correct answer; whether the examination is unfair; whether there is transparency in the examination review process; and whether the examination grading process is arbitrary and capricious.
Findings Of Fact Mr. Santoro took the Examination on November 16, 2018. Petitioner failed the Examination because he scored less than 70 percent correct. The Examination contains 125 questions, 120 of which are scored. The other five are not scored and are considered “pilot” questions for potential use on future examinations. In order to pass the Examination, a candidate must obtain a score of at least 70 percent. All scored questions on the Examination are weighted equally. As a result of failing to pass the Examination, Petitioner was notified of his results. All questions on the Examination had a single correct answer. Cynthia Woodley, Ph.D., employed by Professional Testing, Inc. (“PTI”), as the chief operating officer, is an expert in psychometrics and exam development. She holds a master’s degree in vocational education and a doctorate in curriculum and instruction with a specialization in measurement. Her current position calls for her to manage a number of licensure and certification exam programs. She explained at length how specific questions become part of a professional licensure exam. To develop questions, her company brings in any number of subject matter experts, people actually employed in the professions being tested, and they help develop subject matter questions for a particular exam. That was the process used for development of the Examination in this matter. Once the subject matter experts are trained in exam question writing techniques, they write questions, which are reviewed by other subject matter experts to determine whether the questions are fair and understandable enough to be answered by prospective test takers. Generally, five subject matter experts review each question before it makes its way onto an exam. PTI measures the “P value” of the questions by determining what percent of individuals taking a given exam answer a particular question correctly. For example, a P value of .90 means that 90 percent of the people taking the exam answered a particular question correctly. PTI looks for a wide range of P values in its exam questions. If a P value is too low, say .40, the company might reexamine that question to determine whether it should be removed from future exams since fewer than half the people taking the exam answered it correctly. The business and finance portion of the exam is given to all contractors, regardless of their specialty, with the exception of pool service contractors. Here, Petitioner, a HVAC contractor was administered the same Examination as plumbing contractors, electrical contractors, general contractors, etc. Each of the 120 questions on the exam in this case was equally weighted. There were also five pilot questions inserted into the exam, which did not count towards the total score, but were included as test questions for future exams. Petitioner provided hearsay documents regarding computer hacking and computer glitches associated with some exams administered around the United States. However, he did not connect the articles submitted into evidence to the exam administered in this case or any exam administered by the Department in Florida. Dr. Woodley was familiar with the allegations of computer glitches in testing, but testified that the problems were with K-12 testing in schools, not with professional licensure exams, such as administered by the Department. Therefore, since the hearsay evidence was not linked to the exam at issue or similar professional licensure exams given in Florida, it is entitled to no weight in arriving at the decision in this case. Question BF 1290 has a single correct answer, which is answer “C.” Petitioner selected answer “B.” Petitioner was unable to demonstrate that the answer he selected was correct. Question BF 0473 has a single correct answer, which is answer “A.” Petitioner selected answer “C.” This question asks for an answer of general applicability. Petitioner’s claim that his answer is equally correct is based on a narrow exception in law. Accordingly, Petitioner was not able to demonstrate that the answer he selected was correct. Question BF 0162 has a single correct answer, which is answer “B.” Petitioner selected answer “C.” Petitioner was unable to demonstrate that the answer he selected was correct. Question BF 1691 has a single correct answer, which is answer “C.” Petitioner selected answer “D.” Petitioner was unable to demonstrate that the answer he selected was correct. Petitioner was unable to submit sufficient evidence to show that the Examination is unfair, that there is insufficient transparency in the examination review process, or that the examination grading process is arbitrary and capricious. Accordingly, he cannot prevail in his challenge to the Examination. Petitioner testified that he took and passed the HVAC contractors special license examination on his first attempt. He has taken the Examination on numerous occasions and is yet to be successful. He testified he studied hard for every administration of the exam, but just cannot reach the finish line successfully. While that is unfortunate, the evidence does not support that his failure to succeed on the Examination is the fault of the exam itself or of the Department either in its contracting to have the exam created or in the administration of the exam. From the way he conducted himself at hearing, Petitioner appears to be an intelligent, diligent, and successful individual in his HVAC business. For some unknown reason he has been unable to successfully complete the Examination. His persistence in retaking the Examination multiple times is admirable and should ultimately pay off with his successful passage of the Examination.
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is RECOMMENDED that the Construction Industry Licensing Board enter a final order upholding the Department’s Amended Grade Report finding that Petitioner failed to achieve a passing score on the Construction Business and Finance Examination, which he took on November 16, 2018. DONE AND ENTERED this 23rd day of August, 2019, in Tallahassee, Leon County, Florida. S ROBERT S. COHEN Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 Fax Filing (850) 921-6847 www.doah.state.fl.us Filed with the Clerk of the Division of Administrative Hearings this 23rd day of August, 2019. COPIES FURNISHED: Thomas G. Thomas, Esquire Department of Business and Professional Regulation 2601 Blair Stone Road Tallahassee, Florida 32399-2202 (eServed) Gary Peter Santoro Hometown Air & Services 8229 Blaikie Court Sarasota, Florida 34240-8323 (eServed) Ray Treadwell, General Counsel Office of the General Counsel Department of Business and Professional Regulation 2601 Blair Stone Road Tallahassee, Florida 32399-2202 (eServed) Daniel Biggins, Executive Director Construction Industry Licensing Board Department of Business and Professional Regulation 2601 Blair Stone Road Tallahassee, Florida 32399 (eServed) Halsey Beshears, Secretary Department of Business and Professional Regulation 2601 Blair Stone Road Tallahassee, Florida 32399-2202 (eServed)
The Issue The issue in this case is whether Florida Administrative Code Rule 61G15-21.004(2) is an invalid exercise of delegated legislative authority.
Findings Of Fact The Parties Petitioner is an applicant for licensure as a professional engineer ("P.E.")2 in Florida, and currently works in the discipline of environmental engineering in Florida. His practice focuses primarily on water-related areas within that discipline. Petitioner is not currently licensed as a P.E. Respondent is a board within the Department of Business and Professional Regulation ("Department"). It is the state agency created pursuant to section 471.007, Florida Statutes, and charged with licensing professional engineers in Florida. Respondent is vested with the authority to adopt rules to implement chapter 471, regarding the regulation of the practice of engineering in Florida, as defined in section 471.005(7). Respondent adopted the Challenged Rule at issue in this proceeding. Statutory and Rule Background The engineering profession in Florida is regulated pursuant to chapter 471. A person may become licensed as a P.E. in Florida by applying for licensure, fulfilling specified educational and experience requirements, and 2 For purposes of this Final Order, the terms "professional engineer" or "P.E." will be used to refer to persons who are licensed engineers under chapter 471, Florida Statutes. either being endorsed for licensure as provided in sections 471.015(3) and (5), or passing the required licensure examinations. § 471.015, Fla. Stat. Pursuant to section 471.015, Respondent has adopted Florida Administrative Code Chapter 61G15-20, which codifies, in rule, the requirements for licensure as a P.E. in Florida. An applicant for licensure must be a graduate of a Board-approved engineering program; have the requisite number of years of engineering experience; and have passed the specified licensure exams. Fla. Admin. Code R. 61G15-20.0010. Section 455.217(1)(d) authorizes Respondent to adopt, by rule, the use of a national professional licensing examination that the Department has certified as meeting requirements of national examinations and generally accepted testing standards. To implement section 455.217(1), Respondent has adopted rule 61G15- 21.001, titled "Examination Designated; General Requirements." This rule requires that, unless an applicant qualifies for licensure by endorsement, he or she must pass the National Council of Examiners for Engineering and Surveyors ("NCEES") licensure exam. Part I of the NCEES exam is the Fundamentals of Engineering ("FE") exam, and part II of the NCEES exam is the Principles and Practice Exam ("PP") exam. Respondent has entered into a contract with NCEES to provide the FE and PP exams in Florida. A person must pass both the FE and PP exams to be licensed as a P.E. in Florida. § 471.015(1), Fla. Stat. The Challenged Rule states: "[t]he passing grade for Principles and Practice Exam is determined by the National Council of Examiners for Engineering and Surveying, where psychometric statistical methods are used to determine the level of performance that corresponds with minimal competence in the discipline." Fla. Admin. Code R. 61G15-21.004(2). The Challenged Rule is a subsection of rule 61G15-21.004, which is titled "Passing Grade." The Challenged Rule specifically and exclusively addresses the method for determining the passing grade on the PP exam. Sections 455.217(1)(d) and 471.013 are cited as the rulemaking authority for the Challenged Rule, and sections 455.217(1)(d) and 471.015(1) are cited as the law implemented by the Challenged Rule. The term "engineering," as used in section 471.005(7), includes the term "professional engineering," and defines the types of services and creative work that constitutes "engineering." An "engineer," as defined in section 471.005(5), includes the terms "professional engineer" and "licensed engineer," and means a person who is licensed to engage in the practice of engineering under chapter 471. By contrast, an "engineer intern," as defined in section 471.005(6), means a person who has graduated from a Board-approved engineering curriculum and has passed the FE exam. By definition, these are distinct terms. The term "engineer" is used to describe a person licensed as a P.E. under chapter 471, while the term "engineer intern" is used to described a person who may engage in the kinds of activities described within the term "engineering," as defined in section 471.005(7), but who is not licensed as a P.E. in Florida, and, therefore, is not authorized to hold himself or herself out as a licensed engineer in Florida. The Rule Challenge Petition The Rule Challenge Petition alleges four grounds under section 120.52(8) for invalidating the Challenged Rule. Alleged Invalidity of Challenged Rule under Section 120.52(8)(c) In paragraph 15 of the Rule Challenge Petition, Petitioner asserts that section 455.217(1)(c)—which, at the time the Rule Challenge Petition was filed, was cited as the specific authority for, and law implemented by, the Challenged Rule—did not authorize the Challenged Rule, so that the Challenged Rule enlarged, modified, or contravened the specific provisions of law implemented, pursuant to section 120.52(8)(c). As a result of Petitioner having filed the Rule Challenge Petition, Respondent discovered that it had not updated its citation of the specific authority for, and law implemented by, the Challenged Rule, when section 455.217(1) was amended and renumbered in 1997, so that section 455.217(1)(c) no longer was the correct citation to the law implemented by the Challenged Rule. Respondent requested the Department of State, Administrative Code and Register Section ("DOS"), to make a technical, non-substantive change to the Challenged Rule. As authorized by Florida Administrative Code Rule 1-1.010(10), DOS updated the statutory citation to section 471.217(1)(d), which is the correct citation to the law implemented by the Challenged Rule. This technical change nullifies the alleged invalidity ground set forth in paragraph 15 of the Rule Challenge Petition, and Petitioner concedes this. In paragraph 16 of the Rule Challenge Petition, Petitioner also alleges that the Challenged Rule enlarges, modifies, or contravenes section 455.217(1)(a), because the PP examination does not adequately and reliably measure an applicant's ability to practice the profession regulated by the Department. However, as discussed below, section 455.217(1)(a) is not cited as a specific provision of law implemented by the Challenged Rule, so cannot form the basis of a challenge to the Rule under section 120.52(8)(c). Alleged Invalidity of Challenged Rule under Section 120.52(8)(d) In paragraph 17 of the Rule Challenge Petition, Petitioner alleges that the Challenged Rule is invalid under section 120.52(8)(d) because it is vague, fails to establish adequate standards for agency decisions, or vests unbridled discretion in the agency. In support of this alleged invalidity ground, Petitioner asserts that the Challenged Rule is vague because "the level of performance on the PP exam is stated to correspond with minimal competency, yet there are no rules which provide definitive guidance to NCEES on what constitutes the general areas of competency in regards to engineering practice."3 Petitioner also alleges, in paragraph 19 of the Rule Challenge Petition, that the Challenged Rule is invalid under section 120.52(8)(d) because it equates passage of the PP exam with a 30-year practice experience requirement for licensure by endorsement set forth in section 471.015(5)(b). To this point, Petitioner states: "I can't think of anything more arbitrary than the principles and practice exam equating to near[-]retirement level experience."4 Alleged Invalidity of Challenged Rule under Section 120.52(8)(e) In paragraph 18 of the Rule Challenge Petition, Petitioner alleges that the Challenged Rule is invalid pursuant to section 120.52(8)(e) because it is arbitrary, for several reasons. Paraphrased, these reasons are: passage of the PP exam does not accurately reflect, or equate to, minimal competence in the discipline; the PP Exam does not accurately evaluate an individual's engineering ability level, but instead evaluates an individual's exam performance compared to average group exam performance; the PP exam does not reliably distinguish between minimal competence and incompetence to practice engineering, as evidenced by the fact that engineers who fail the PP exam still competently perform, and, thus, keep, their engineering jobs; passing the PP exam, by itself, does not certify an individual to competently perform any engineering service or creative work as defined in 3 Section 455.217(1)(b) requires, for each exam developed by the Department or a contracted vendor, that the general areas of competency covered by the exam be specified by rule. The last sentence of that subsection states that the requirements of subsection (b) do not apply to national exams, such as the NCEES PP exam, which are approved and administered pursuant to section 455.217(1)(d). Thus, the law implemented by the Challenged Rule does not require areas of competency to be specified in the Challenged Rule. 4 Because paragraph 19 of the Rule Challenge Petition alleges that the rule is arbitrary, the undersigned considers this paragraph to constitute a challenge to the Challenged Rule under section 120.52(8)(e), rather than under section 120.52(8)(d), as cited in the Rule Challenge Petition, and has addressed this ground in the Conclusions of Law section dealing with that alleged invalidity ground. section 471.005(7); the PP exam does not reliably determine if an examinee is minimally competent, due to an incorrect reference point; and the PP exam does not reliably distinguish between individuals whose practice of engineering would protect the public health and safety and those whose practice of engineering would constitute a danger to public health and safety. Alleged Invalidity of Challenged Rule under Section 120.52(8)(f) In paragraph 20 of the Rule Challenge Petition, Petitioner alleges that the Challenged Rule is invalid pursuant to section 120.52(8)(f) because it imposes regulatory costs on the regulated person, county, or city, which could be reduced by the adoption of less costly alternatives that substantially accomplish the statutory objectives. To this point, Petitioner proposes a state- conducted investigation of an applicant as a substitute for the PP exam, and contends that "there's a premium associated with a national exam. It's also clear that the regulatory cost imposed on me and all future examinees could be substantially reduced if the Board conducted their exam as an investigation and did not incorporate the NCEES exam." However, as more fully discussed below, this challenge ground is time- barred by section 120.541(1)(g), and, therefore, is not a legally-cognizable basis for invalidating the Challenged Rule in this proceeding. The Parties' Stipulated Facts Petitioner is an Engineer Extern, Texas EIT 56990. Pursuant to section 471.015(1), the Florida Engineers Management Corporation "shall issue a license to any applicant who the Board certifies is qualified to practice engineering and who has passed the [FE] exam and the [PP] exam." Both the FE exam and the PP exam are created by NCEES, pursuant to section 455.217(1)(d), which states, in pertinent part: "a board . . . may approve by rule the use of any national examination which the [Department of Business and Professional Regulation] has certified as meeting the requirements of national examinations and generally accepted testing standards pursuant to department rules." Pursuant to section 471.015(1), Petitioner passed the FE exam on September 3, 2016. Passing the FE is a prerequisite to take the PP exam. Petitioner registered to take the NCEES computer-based PP exam for environmental engineering on April 22, 2020. He paid the $350 exam fee and additional monies for test preparation material, and spent at least 100 hours preparing for the exam. He was notified, on April 6, 2020, by Pearson Vue, the test center company, that the exam was cancelled due to the Covid-19 pandemic. On April 7, 2020, Petitioner registered to take the exam on July 15, 2020, which was the earliest available date for taking the exam in his local area. On April 9, 2020, Petitioner canceled his registration for the July 15, 2020, exam and decided to apply for licensure as a P.E. without passing the PP exam. Pursuant to section 471.015(2)(a)1., on May 18, 2020, Petitioner filed his application for licensure with the Board. The application provides information stating that Petitioner meets the requirements of section 471.013(1)(a)1., and has at least four years of active engineering experience of a character indicating competence to be in responsible charge. In compliance with the education requirements of section 471.013(1)(a)1., Petitioner earned a Bachelor of Science degree in Chemical Engineering from the University of Florida, which is accredited by ABET. Petitioner provided an official transcript to the Board for verification purposes. In compliance with rule 61G15-20.002, Petitioner states in his application that he has at least four years of acceptable engineering practice, consisting of one year of experience equivalent through his Master of Engineering degree from the University of Texas at Austin, an ABET- accredited institution; and over three years of professional engineering work verified by licensed engineers. According to rule 61G15-20.002, an applicant must list three current personal references who are professional engineers. Thirteen licensed engineers submitted documentation to the Board regarding Petitioner's type of qualifying experience, level of engineering competency, and professional integrity. All references circled "yes" to the question "would you employ the applicant in a position of trust?" Two additional references in the engineering industry also provided their opinion on Petitioner's integrity and competence. On June 1, 2020, Petitioner received notification from the Board that his Florida 0901 1031-P.E. Endorsement application was incomplete. Petitioner addressed all items listed in a timely fashion to participate in the June 10, 2020, Board meeting. The Board stated during an informal hearing on June 10, 2020, that Petitioner would not be granted P.E. licensure due to not having passed the PP Exam. Petitioner is substantially affected by the Challenged Rule because it disqualifies him from becoming a licensed engineer in Florida without passing the PP exam. Findings of Fact Based on Evidence Presented at the Final Hearing A. Findings Regarding the Evidence Presented in Petitioner's Case Testimony of Witnesses Hoot and Grace David Hoot and Nigel Grace, both of whom are licensed professional engineers, testified regarding Petitioner's abilities and skills as a practicing engineer. Neither Hoot nor Grace was qualified as an expert in the field of psychometrics or related topics. Therefore, any opinions regarding these subject matters to which they testified at the hearing have been treated as personal opinions, rather than expert opinions. Hoot characterized Petitioner as a good, diligent young engineer, and described Petitioner's role in various projects on which they worked together. Based on his work with Petitioner, it was Hoot's personal opinion that Petitioner possesses the integrity and competence to work as a licensed engineer who would serve the public health and safety. Hoot testified that when he took the PP exam approximately 38 years ago, it was a free response exam. He stated that he was "not exactly sure" that a multiple-choice exam captures an examinee's ability to apply reason and judgment, but he acknowledged that he does not know how the PP exam currently is developed. He offered his personal opinion that it was possible for a competent engineer to fail the PP exam. Hoot also offered his personal view that engineering licensure constitutes a standard of care; however, he did not think anything provides a guarantee of competence. It was his personal view that experience tended to make one more competent. He also offered his personal view that as an engineer gains more experience and becomes more specialized, it is understandable that he or she would not score as well as an engineer who is gearing up toward the four-year experience goal which enables them to apply for licensure as a P.E. As Hoot put it, "life happens. You have children involved. You move jobs. I think you . . . maybe have less time to study. . . you get farther away from the study habits of . . . learning to be able to take tests." Grace, who is employed as a P.E. with Brown and Caldwell ("B&C"), a large U.S. engineering firm, testified regarding Petitioner's work experience while he was employed by B&C. Petitioner's experience included working on drinking water projects, utilities, upgrading process equipment, site evaluations, bench scale testing, and other projects. Based on Grace's work with Petitioner, it was his personal opinion that Petitioner possesses the integrity and competence to work as a licensed engineer who would serve the public health and safety. Grace took the PP exam approximately 28 years ago, and at the time, a major portion of the exam consisted of long-form written exam questions that provided the opportunity for examinees to provide free response answers and earn partial credit for partially-correct answers. He testified that the exam also had a multiple-choice component. Grace testified that, "based on instinct," he knows well-designed multiple-choice questions can provide the same insight into an examinee's decision-making judgment as long-form questions. Grace's personal view is that passing the PP exam does not guarantee competence, and it is possible for an engineer to be competent in some engineering disciplines but not others. Grace also agreed that it was logical that an examinee with approximately four years' experience would perform better on the PP exam because he or she would have better-honed test-taking skills and be fresher in some areas tested on the exam. By contrast, engineers who have practiced longer have more experience, but often have become specialized and further removed from the test-taking environment. Thus, passing a broad-spectrum exam becomes a bigger hurdle for engineers who have practiced longer. Petitioner's Testimony Petitioner testified on his own behalf at the final hearing. As discussed above, Petitioner holds a bachelor of science degree in chemical engineering from an ABET-accredited institution, and holds a master of engineering degree from an ABET-accredited institution. Although Petitioner, through his training as an engineer, is skilled at mathematics and statistics, he is not trained, and does not have any substantial experience, in the field of psychometrics.5 5 As discussed in greater detail below, psychometrics is a specialized field of study that concerns the theory and technique of objective psychological measurement of skills, knowledge, abilities, and educational achievement. Petitioner acknowledged that he does not have a degree in psychometrics and that he is not trained in, or knowledgeable about, preparing and administering high-stakes professional examinations. At the time he filed the Rule Challenge Petition, Petitioner had no personal knowledge of the work done by psychometricians; did not know what a "cut score" was; and was not familiar with the Modified Angoff Method, item response theory, the specifics of converting raw scores to scaled scores, or other any psychometric tools and methods employed to prepare and score the PP exam. Petitioner acquired some rudimentary knowledge in a few of these areas in preparing for the final hearing in this proceeding.6 Section 90.701(2), Florida Statutes, prohibits a lay witness from testifying as to an opinion regarding a matter involving specialized knowledge.7 Accordingly, it is determined that Petitioner is not competent to provide an expert opinion regarding psychometrics and related areas, such as item response theory. He was not qualified, tendered, or accepted at the final hearing as an expert in psychometrics or any related areas. Because Petitioner was not qualified to testify as an expert witness at the final hearing, Petitioner's testimony regarding psychometrics, item response theory, scaled scores, the Modified Angoff Method, high-stakes professional testing, and all other specialized subject matters, consisted of opinion testimony by a lay witness. The only instances in which a lay witness 6 Petitioner does not have any special knowledge, formal training, education, or experience in the specialized field of psychometrics. His knowledge about these areas was acquired by reading and study in preparation for the final hearing. Because Petitioner lacks special knowledge, experience, training, and education in psychometrics, he is not qualified to testify as an expert in psychometrics or related topics, such as item response theory. See Chavez v. State, 12 So. 3d 199, 205 (Fla 2009)(in determining whether a witness is qualified to render an opinion as an expert in a specialized field, the court must determine whether the witness is adequately qualified to render an opinion based on special knowledge, experience, training, or education). 7 Chesser v. State, 30 So. 3d 625 (Fla. 1st DCA 2010)(it is error for a court to accept opinion testimony of a lay witness in a specialized subject matter area). may provide opinion testimony are when the lay witness's opinion is based on firsthand knowledge through personal perception.8 As the undersigned explained during the final hearing, rather than excluding Petitioner's lay opinion testimony, she would consider these pertinent evidentiary principles in determining the weight to be afforded Petitioner's lay opinion testimony in this proceeding. Applying these evidentiary principles, it is determined that Petitioner's testimony regarding psychometrics and related topics, such as item response theory, classical response theory, and high-stakes test reliability and validity, concerned specialized subject matters not within the realm of common knowledge or based on Petitioner's personal perception. Rather, such specialized subject matters required expert witness testimony, pursuant to section 90.702, and as addressed above, Petitioner was not shown to be an expert in any of these specialized subject matters. Because Petitioner's testimony constituted the type of opinion testimony that is not permissible by a lay witness, pursuant to section 90.701, such testimony is not afforded weight in this proceeding. Petitioner testified that "minimal competence," which is the standard measured on the PP Exam, equates to "competence" as defined in the dictionary—that is, the "sufficient ability for a specific need or requirement." However, this position ignores that, for purposes of the Challenged Rule, "minimal competence" is a term of art specifically used, in the psychometric measurement context, to describe the level of competence that corresponds to a passing score on the PP exam. As more fully discussed below, the PP exam is developed and scored using psychometric tools and methods. 8 Nat'l Commc'ns. Indus., Inc. v. Tarlini, 367 So. 2d 670, 671 (Fla. 1st DCA 1979)(lay witness testimony regarding a specialized subject matter was not admissible into evidence because the testimony was not regarding a subject matter about which the witness could testify based on common knowledge or his personal perception.) Petitioner contends that engineering experience is, by itself, a reliable measure of competence, so it is unnecessary to also pass the PP exam. Petitioner testified "the application process and [PP] exam have two entirely different methods to identify the same result: whether an engineer in training is competent enough to become a licensed engineer It is not logical for two checks of competence to come up with different results. There should be consistency." In support of this position, Petitioner relies on section 471.015(2)(a), which requires "at least 4 years of active engineering experience of a character indicating competence to be in 'responsible charge' of engineering." § 471.015(2)(a), Fla. Stat. "Responsible charge" is defined in rule 61G15-18.011(1) as the degree of control an engineer is required to maintain over engineering decisions made personally or by others over which the engineer exercises supervisory direction and control authority. An engineer in responsible charge is the "engineer of record," as defined in rule 61G15-30.002(1). Rule 61G15-30.002(1) defines "engineer of record" as a Florida professional engineer who is in responsible charge. Thus, an engineer who is qualified, for purposes of being in responsible charge pursuant to section 471.015(2)(a) must, in addition to having the minimum statutory experience, be a licensed P.E. This means that he or she necessarily must have passed the PP examination. These statutory and rule provisions collectively reinforce the point that for an engineer to demonstrate competence for purposes of holding himself or herself out as an "engineer," as defined in section 471.005(7), he or she must satisfy all three requirements of section 471.015(2)(a)—i.e., education, experience, and passing the licensing exam.9 9 The requirement to meet these three requirements, including the PP exam, is codified in section 471.051(2)(a). Eliminating the exam requirement and relying strictly on education and/or experience for licensure would require the Florida Legislature to amend this statute Petitioner echoed the testimony of Hoot and Grace that licensure is not a guarantee of competence, and that passing the PP exam does not guarantee minimal competence. To this point, he testified that he does not believe that the PP exam adequately and reliably measures an applicant's ability to practice engineering, and that experience is a better indicator of competence than passing the exam. By way of example, Petitioner described his own experience10—which he characterized as "directly matching" the activities in which a licensed engineer engages—and compared that experience to measuring competence by an exam, which Petitioner characterized as "attempt[ing] to indirectly measure my ability as an engineer." Based on his personal experience, Petitioner contends that experience better demonstrates competence to be licensed as a P.E.; that passing the PP exam does not indicate minimal competence to practice engineering; and that failing the PP exam does not mean that the examinee is not minimally competent. He further testified that examinees who fail the PP exam likely are minimally competent, since the engineering jobs they hold when applying for licensure likely would require that they be minimally competent in order to have been hired.11 to eliminate the exam requirement. The undersigned is not authorized by statute or the Florida Constitution to eliminate the PP exam requirement for licensure under chapter 471. 10 Petitioner's experience, set forth in his P.E. licensure application, was verified by his supervising engineers. 11 Petitioner appears to conflate being determined not "minimally competent" for purposes of passing the PP exam, with "incompetence," which is defined in Florida Administrative Code Rule 61G15-19.001(5) as the "physical or mental incapacity or inability of a professional engineer to perform the duties normally required of the professional engineer." Part of this confusion may be due, in part, to Respondent's response to one of Petitioner's interrogatories asking for a definition of "minimal competence." Rather than directly answering the interrogatory, Respondent referred Petitioner to the definition of "incompetence" for purposes of imposing discipline under Respondent's disciplinary rules—thus causing Petitioner to understandably assume that failing to demonstrate minimal competence through passing the PP exam equates to "incompetence," as defined in rule 61G15-19.001(5). However, the fact that Petitioner has not demonstrated "minimal competence" on the PP exam does not mean that he is incompetent; it simply means that he has not yet passed the PP exam for licensure as a P.E. in Florida. To this point, if failing to demonstrate "minimal competence" by passing the PP exam equated to being incompetence, every person who performs engineering work in To further illustrate this point, Petitioner noted that the data regarding passage rate of the PP exam shows that examinees having zero years of experience are almost twice as likely to pass the PP exam as examinees having 11 or more years of experience. However, as Hood and Grace explained, and as further discussed below, engineers having more than four to five years of experience begin to specialize in narrower fields and "life happens," in that personal and professional circumstances render it more difficult to prepare for and take a high-stakes test. Petitioner also disputed the accuracy of the PP exam preparation and scoring process. In particular, he took issue with the "model law engineer" standard to which the exam is designed. As discussed more extensively below, this standard equates to the competence level of an engineer having four years of engineering experience and who is capable of practicing engineering in a manner that protects the public health and safety. In particular, Petitioner contends that designing the PP exam to the "model law engineer" standard is unfair to anyone taking the exam that does not have exactly four of years of engineering experience. Notably, however, section 471.015(1), which is the statute implemented by the Challenged Rule, establishes four years as the engineering experience required for licensure as a P.E. Thus, the "model law engineer" standard is rationally related to the statutory minimum experience level for purposes of demonstrating minimum competency to be licensed. Petitioner also contends that the subject matters tested on the PP exam are unfairly broad, so that engineers who specialize in a particular area within an engineering discipline—such as specializing in water-related areas in environmental engineering—are disadvantaged by being required to take Florida but has not passed the PP exam would be "incompetent," and, thus, potentially subject to disciplinary action. an exam that covers a broad range of areas beyond his or her area of specialty. Petitioner further contends that it is irrational to test an examinee on particular areas that are irrelevant to his or her work and/or desired career path. However, the PP exam for a particular discipline is specifically designed to ensure that a licensed P.E. is competent to practice over a range of specific areas encompassed within that particular discipline. This is because once a person becomes a licensed P.E., he or she may practice engineering within any discipline or specific area within that particular discipline, subject to the professional and ethical requirements to limit practice to the disciplines and areas in which the engineer is actually competent. Thus, the breadth of the PP exam is designed to help ensure minimal competence to practice engineering in a manner that protects the public health and safety. Petitioner also contends that because the PP exam for some engineering disciplines tests a broader range of areas than the PP exam may test for other disciplines, the exam inconsistently measures minimal competency across the range of engineering disciplines. However, as discussed in detail below, the subject matters tested on the PP exam for a given discipline are chosen by subject-matter experts who are licensed engineers practicing in that particular engineering discipline, and are deemed, by those subject matter experts, to be most important to test for purposes of measuring competency in that discipline. Thus, while the number of discrete subject matters tested on the PP exams may differ across the various engineering disciplines, this difference is, factually and logically, a function of expert consensus regarding which subject matters need to tested to demonstrate minimal competence. Petitioner also contends that the Challenged Rule is vague because it does not specifically identify the disciplines, and the areas within each discipline, that are tested on the PP exam. Respondent has contracted with NCEES to be responsible for preparing, administering, and scoring the PP exams, pursuant to section 455.217(1)(d). NCEES conducts a methodical process, discussed in detail below, to determine the specific disciplines for which to develop a PP exam and the areas to be covered on the PP exam for a discipline. Exam specifications are then developed by subject matter experts within that discipline, and are published by NCEES. These specifications inform prospective examinees regarding the particular areas that will be tested on the PP exam for the discipline, and the number of questions for each specific area that will appear on the exam. Thus, prospective examinees are not left to wonder or guess about which disciplines will be tested; the areas within each discipline that will be tested; or the relative weight that will be assigned to each area tested.12 Although the Challenged Rule does not identify the specific disciplines tested on the PP exam, rule 61G15-21.001(1)(b)—which actually adopts the PP exam as an engineering licensure exam in Florida—states that the PP exam "is given by discipline." Therefore, even if section 455.217(1)(d) required the specific areas of competency to be identified by rule, such areas would have been identified in rule 61G15-21.001(1), rather than in the Challenged Rule.13 12 See Cole Vision v. Dep't of Bus. And Prof'l Reg., 688 So. 2d 404, 410 (Fla. 1st DCA 1997)(a rule is impermissibly vague if it is drafted in terms so vague that men of common intelligence must necessarily guess at its meaning or application. 13 Neither sections 455.217(1)(d) nor 471.015(1) specifically authorize or require Respondent to adopt rules identifying the general areas of competency tested on the PP exam. By contrast, exams developed by the agency pursuant to section 455.217(1)(b) must identify, by rule, the general areas of competency to be tested. Had the Legislature intended for exams authorized under section 455.217(1)(d)—of which the PP exam is an example—to adhere to the same requirement, the statute would have so stated. See Pro-Art Dental Lab, Inc. v. V- Strategic Grp., LLC, 986 So. 2d 1244, 1258 (Fla. 2008)(the specific mention of one thing in a statute implies the exclusion of another). Furthermore, section 120.54(1)(g) expressly requires a rule to address only one subject. Thus, if the Challenged Rule also addressed the areas of competency to be tested on the PP exam, it would violate section 120.54(1)(g). Petitioner also asserted, at the final hearing, that Respondent did not certify the PP exam as meeting the requirements of national examinations and generally accepted testing standards pursuant to department rules, as required by section 455.217(1)(d). However, Petitioner did not raise this alleged invalidity basis in the Rule Challenge Petition, so he is foreclosed from raising and litigating it at the final hearing. See § 120.56(1)(b), Fla. Stat. Petitioner also testified that, in transitioning from paper-and-pencil PP exams to computer-based exams, NCEES is relying on two different theories—classical test theory and item response theory—and that this reliance does not comport with generally accepted testing standards. However, as discussed above, Petitioner was not qualified as an expert in the specialized area of high-stakes examination preparation and scoring; thus, his testimony constitutes lay opinion regarding this specialized subject matter. He did not present any competent substantial evidence to support his contention that the PP exam does not meet generally accepted testing standards.14 Petitioner also testified that item response theory, which is a psychometric tool used in developing and scoring the PP exam, is an invalid means of determining the competence of an engineer. To this point, Petitioner testified that the "model law engineer" is not a real person, but is instead an imaginary person created by subject matter experts to define what a minimally competent engineer should know. Thus, according to Petitioner, 14 Because Petitioner was not qualified, tendered, or accepted as an expert in these specialized subject matters, and because his testimony on these matters consists of inadmissible lay opinion testimony, this testimony has not been afforded weight. §§ 90.701 and 90.702, Fla. Stat. the model law engineer standard is the wrong reference point for determining minimal competency to practice engineering.15 Petitioner also testified that the PP exam does not accurately measure ability, which is a latent trait for which an arbitrary measurement scale must be created. He testified that the model law engineer standard is the midpoint of this scale, and that the purpose of the scale is to determine whether examinees fall above or below that midpoint.16 He further contended that the PP exam does not accurately measure ability, because performance on the exam may be influenced by extraneous variables, such as test anxiety. Petitioner also testified regarding item response theory, which, as previously noted, is a psychometric tool used in developing and scoring high- stakes exams—a subject about which Petitioner had no training in, or knowledge about, until he prepared for the final hearing in this proceeding. Specifically, Petitioner testified that the item characteristic curve is the basic building block of item response theory, and that there are two technical properties of an item characteristic curve: difficulty of the item, and the ability of the item to discriminate between examinees' abilities. Petitioner testified that another basic principle of item response theory is that the examinee's ability is a variable with respect to the items used to determine it. According to Petitioner, this principle rests on two 15 Petitioner's testimony on this point was based on excerpts from a book titled Item Response Theory and a book titled The End of Average. The Item Response Theory book is a treatise on psychometrics, a highly specialized field about which Petitioner was not qualified to testify as an expert, and which is not susceptible to lay witness opinion testimony. Thus, Petitioner's testimony on these points is not assigned weight. See §§ 90.701 and 90.702, Fla. Stat. Additionally, excerpts from The End of Average were determined irrelevant, so were not admitted into evidence. 16 As support for this testimony, Petitioner selectively cited and quoted the deposition testimony of Timothy Miller, Respondent's expert on the development and scoring of NCEES's psychometric-based PP exams. The specific context of Miller's deposition testimony was that when an exam item is overexposed, it is subject to drift, which means that the percentage of correct answers for the item increases to the point that the item no longer is a good discriminator. As further discussed below, Petitioner's testimony on this point was directly and persuasively countered by Miller's expert testimony regarding scaled scores and setting the passing score for the PP exam. conditions: that all items measure the same underlying latent trait, and the values of all item parameters are in a common metric. According to Petitioner, this principle reflects that the item characteristic curve spans the entire ability scale; thus, the practical implication is that a test located anywhere along the ability scale can be used to estimate an examinee's ability, such that an examinee could take a test that is easy or hard, and on average, would score at the same estimated ability level. Petitioner testified that this stands in contrast to classical test theory, which he contends is a better discriminator of examinee ability.17 Petitioner also testified that the psychometric methods used to develop and score the computer-based PP exams are flawed because "difficulty" is subjective and entirely dependent on the individuals developing the PP exam. Thus, according to Petitioner, in scoring a computer-based multiple-choice PP exam, it is impossible to know whether a particular examinee got the answer right due to a reasonable approach in answering the question, or by guessing. Petitioner contends that for this reason, multiple-choice test questions developed using item response theory are not good discriminators of examinees' ability; thus, even if an examinee does not correctly answer enough questions to pass the exam, that does not mean that the examinee is not knowledgeable in that area.18 Petitioner further testified that because difficulty is a subjective parameter, different forms of the PP exam inherently have different levels of difficulty. Thus, according to Petitioner, it is a matter of luck whether an examinee takes a more difficult form or an easier form of the exam. Further to this point, Petitioner testified that because an examinee does not take multiple forms of the exam, but instead takes only one form, the determination of the examinee's ability is solely dependent on a subjective 17 Refer to note 15, supra. 18 Refer to note 15, supra. parameter—i.e., the difficulty of the test questions as determined by subject matter experts. Petitioner contends that, as a result, the PP exam does not accurately measure an examinee's ability, and, therefore, is not a valid exam.19 Petitioner also testified that because statistical indices of reliability and validity are not attributes of an exam, a researcher may select what seems to be an appropriate test for his or her purposes, when, in fact, the selected test does not have any level of reliability or validity. Thus, Petitioner testified, reliability and validity are values that reside in test scoring, not in the test itself. Petitioner testified that validation, in statistics, is the process of accumulating evidence that supports the appropriateness of the inferences that are made of student responses for assessment uses. He testified that validity refers to the degree to which the evidence indicates these interpretations are correct and the manner in which the interpretations are used is appropriate.20 Petitioner testified regarding three types of validity evidence: content, construct, and criterion evidence.21 Specifically, Petitioner testified that content evidence refers to the extent to which an examinee's responses to a given assessment reflect the examinee's knowledge of the content being tested; thus, to the extent an exam inadvertently measures a parameter that is not related to the examinee's knowledge of the content being tested, it is invalid. 19 Refer to note 15, supra. 20 Petitioner's testimony relied on, or was paraphrased from, a document titled The Scoring Rubric Development. Again, because this topic and document address a matter within the specialized fields of psychometrics, high-stakes testing, and test-scoring statistics, which are areas in which Petitioner was not qualified as an expert, and which are not susceptible to lay opinion testimony, pursuant to sections 90.701 and 90.702, Petitioner's testimony relying on this document, including his testimony regarding content, construct, and criterion-related evidence, is not afforded weight. 21 Refer to note 15, supra. This determination regarding the weight afforded Petitioner's testimony applies to paragraphs 94 through 100 herein. According to Petitioner, the content-related evidence for the PP exam for each discipline is inconsistent, so that the PP exam for a given discipline does not accurately measure minimal competence for that discipline. Petitioner also testified that the weighting of different topics on the PP exam necessarily creates an advantage for engineers who work in areas more heavily weighted on the exam, while creating a disadvantage for engineers who work in areas that are less heavily weighted on the exam. Petitioner also testified that to accurately determine minimal competence in all engineers, the model law engineer standard should be keyed to, and the content tested on the exam should be directed toward all engineers, including those having more than four years of experience. Petitioner also testified that construct-related evidence consists of external benchmarks, such as results and explanations, of internal evidence of psychological processes, such as reasoning. Petitioner testified that because multiple-choice exams do not provide evidence of an individual's reasoning process, they do not generate construct-related evidence for purposes of determining exam validity. Petitioner testified that free response paper-and pencil-exams provide construct-related evidence, so are better indicators of an examinee's knowledge. Petitioner also testified regarding criterion-related evidence, which relates to the extent to which the results of an assessment, such as the PP exam, correlate with a current or future event. By way of illustration, Petitioner testified that criterion-related evidence considers the extent to which a student's performance may be generalized to other relevant areas. Petitioner testified that an examinee's performance on the PP exam is not generalizable to other relevant activities, so it is impossible to determine whether the exam actually corresponds to minimal competence in the workplace. In sum, Petitioner testified that the PP exam does not meet content, construct, or criterion-based evidence for purposes of determining whether it is a valid exam. Thus, Petitioner reasons, it is logical to conclude that because the PP exam is not a good discriminator between minimally competent and incompetent engineers, it does not reliably and adequately measure competence. Petitioner also testified that because passing the PP exam is only one component of licensure, it fails to meet criterion-based validity, in that the exam, by itself, does not certify a passing examinee to practice as a P.E. As Petitioner put it, "you're just passing the exam as part of the requirement for licensure." Petitioner reasoned that if passing the PP exam corresponds to minimal competence, the experience and education requirements of section 471.015(2)(a) are redundant. Also to this point, Petitioner testified that the preapproval process to take the PP exam is directly related to an examinee's actual work experience as an engineer, while taking the exam merely entails answering questions about engineering work. Thus, Petitioner contends, a competent engineer, as shown through Respondent's preapproval process, may nonetheless fail the exam. Petitioner asserts that this further shows that the exam does not accurately measure minimal competence. Petitioner also testified that, in his view, delaying licensure of potentially competent engineers due to postponing the exam due to the Covid-19 pandemic does not serve the interest of public health and safety. To that point, he testified that the inability to obtain a variance, which would relieve examinees from having to take and pass the exam under such circumstances, renders the Challenged Rule arbitrary.22 Petitioner also contended that passing the PP exam should not be required, because other engineering professional associations—specifically, the European Federation of National Engineering Associations ("FEANI"), which represents engineers in European countries—allow licensure through 22 Petitioner's point regarding inability to obtain a variance or waiver is addressed in the Conclusions of Law. education and experience requirement, without requiring a professional exam to be taken and passed. However, because section 471.015, which governs the licensure of engineers in the state of Florida, requires a professional licensure examination to be taken and passed as part of the P.E. licensure requirements, Petitioner's testimony and argument regarding FEANI's practices and requirements take issue with the statute, rather than the Challenged Rule, and, thus are irrelevant. Petitioner also contends that the examination fee for the PP exam is arbitrarily set, rendering the Challenged Rule arbitrary. However, as discussed above, the Challenged Rule only addresses determining the passing grade for the PP exam using psychometric methods. The Challenged Rule has nothing to do with establishing or setting an examination fee. Thus, this challenge ground has no basis in fact or law.23 C. Findings Regarding the Evidence Presented in Respondent's Case Respondent's Expert Witnesses Respondent presented the testimony of Timothy Miller, P.E., who serves as Director of Examination Services for NCEES. Miller has held this position for approximately 15 years. His job-related activities and responsibilities include directing exam development, publication, scoring, and fulfillment of the licensing exams for engineers and surveyors; coordinating exam development committees consisting of over 800 volunteer subject matter experts who work on developing each NCEES exam; overseeing the exam development process and providing advice and guidance regarding engineering exam development, administration, production, scoring, analysis, and reporting; serving as a testing process consultant to exam development committees; and other exam-development and administration-related matters. 23 Additionally, this challenge ground was not raised in the Rule Challenge Petition, so is not at issue in this proceeding. See § 120.56(1)(b), Fla. Stat. Before Miller was promoted to his current position, he served as an examination development engineer for NCEES. In that position, Miller was responsible for planning and coordinating engineering exam development, production, administration, scoring analysis, and reporting for certain assigned examinations; serving as a testing consultant working with engineering exam development committee chairs regarding quality and number of exam development volunteers; and overseeing development and administration of the licensing exams in the specific fields of environmental controls systems, metallurgical engineering, and mechanical engineering. Through his experience in these positions with NCEES, Miller is an expert in professional examination development and scoring, particularly with respect to the development and scoring of the NCEES FE and PP examinations. Before being employed with NCEES, Miller practiced civil and structural engineering with several private-sector engineering firms. He has been a professional engineer since 1984, and is licensed as a P.E. in South Carolina, North Carolina, Maryland, Delaware, Pennsylvania, and New Jersey. Respondent also presented the testimony of Dr. Michelle Rodenberry, P.E., an associate dean and associate professor at the Florida A&M University–Florida State University College of Engineering. Her engineering expertise is in the field of structural engineering—specifically, bridge engineering. Rodenberry was appointed to the Board in 2012, and she served as a Board member until 2018. She is now an emeritus Board member. While on the Board, she served as chair of the education committee, and was involved in reviewing applications for licensure as a P.E. in Florida. Development, Scoring, and Validation of the PP Exam The NCEES engineering exams are national licensing exams that are recognized by every engineering licensing entity in each of the U.S. states, as well as by the engineering licensing entities in Washington, D.C.; Puerto Rico; the U.S. Virgin Islands; and the other U.S. territories and protectorates. There are approximately 26 different engineering disciplines, each of which is tested by a separate PP exam specific to that discipline. In the 1990s, NCEES decided to transition from subjectively-graded pencil-and-paper examinations to an objectively-graded computer-based multiple-choice exam format. Currently, approximately one-third of the PP exams, including environmental engineering, have been converted to a computer-based format, and all but one of the exams in the other disciplines is in the process of being converted. The reason NCEES is transitioning the PP exam from a pencil-and- paper format to a computer-based format consisting of multiple-choice questions is to help eliminate subjectivity in grading, so that the exam papers are consistently graded across groups of examinees. Additionally, a computer-based format consisting of objective multiple-choice questions allows the difficulty of the exam to be psychometrically evaluated for purposes of determining the passing score for a particular administration of the exam. To that point, because computer- based multiple-choice exams are objectively scored, exams offered at different times during the year are able to be compared, or equated, for purposes of setting the passing grade for a particular exam administration.24 Respondent entered into a contract with NCEES in 2009, pursuant to which NCEES provides the FE and PP exams for engineer licensure in Florida. In 2013, the contract was amended to allow NCEES to provide the exams by computer-based testing, using Pearson Vue as its exam 24 As Miller explained,"[i]f they were different on a difficulty level, the harder exam, the standard would actually be lowered so that it would be fair across administration so everybody was treated consistently. Or if my exam was less difficult, the standard would be raised. I would have to get more questions right." administering entity. The FE and PP exams are administered by Pearson Vue at its testing centers. NCEES develops model laws and rules that represent best practices with respect to state licensing of engineers. The aim of these model laws and rules is to achieve uniformity and consistency throughout the states and the U.S. territories and protectorates in the licensure of professional engineers. A significant benefit of such consistency and uniformity is the resulting "mobility" for licensed professional engineers—that is, the ability to more easily become licensed to practice engineering in multiple states. The NCEES model laws and rules establish the "model law engineer," which defines and constitutes the standard for minimal competence in a specific engineering discipline for purposes of being licensed as a P.E. in that discipline. The model laws and rules define the "model law engineer" as a person who holds a degree from an engineering educational program accredited by ABET, has four years of active engineering practice experience, and passes the FE and PP exams. The model law engineer standard equates to the competence level of an engineer having four years of engineering experience who is capable of practicing engineering in a manner that protects the public health and safety. This constitutes the minimum competence level that an applicant must demonstrate for purposes of being licensed as a P.E.25 in the 50 states and the U.S. territories and protectorates. Thus, the NCEES PP exam is constructed to test engineering ability keyed to the model law engineer standard. That is, the PP exam is designed to determine the ability level of an applicant for P.E. licensure for purposes of 25 Refer to note 12, supra. The term "minimal competence," as used in the Challenged Rule, is specifically keyed to the "model law engineer" standard for purposes of being licensed as a P.E. It is not meant to indicate or imply that an engineer who does not take or pass the PP exam is per se incompetent, such that he or she is not competent to engage in work constituting engineering, as defined in section 471.005(7). comparing that ability level to that of an engineer having four years' experience who is able to practice engineering in a manner that protects the public health and safety. Examinees having four years of engineering experience after graduation have the highest pass rates on the PP exams. Pass rates for examinees with more or less than four years of experience are lower, typically in proportion to the length of time before or after the four-year experience mark when they take the PP exam. Miller explained, credibly and persuasively, that the reason for the drop-off in PP exam performance after the four-year mark is that "life happens." Engineers gain more experience, and many become specialized in a relatively narrow niche, or move into managerial, non-technical positions. Additionally, because the PP exam does, in part, test subjects that one learns in college, the longer an examinee is out of college, the less subject matter recall in certain areas he or she may have. "Psychometrics" is the specialized field of study concerned with the theory and technique of psychological measurement. Specifically, psychometrics entails the objective measurement of skills and knowledge, abilities, and educational achievement. Among other specialized areas of practice, psychometricians focus on the construction and validation of assessment instruments, and theories, such as item response theory, that relate to psychological measurement. Psychometricians typically have graduate training and all possess specialized qualifications that enable them to engage in objective psychological measurement. PP exams are designed to determine minimal competence in a specific engineering discipline. "Minimal competence" is the minimal amount of knowledge required to practice in that particular engineering discipline in order to protect the public health and safety. For any specific engineering discipline for which it has been determined that a PP exam should be given,26 there is an approximately three-year due diligence period in which subject matter experts in that discipline work to determine the topics that should be tested on the exam. The PP exam for each specific discipline is developed by subject matter experts, who volunteer and meet on a monthly basis to develop, review, and evaluate the questions for the PP exam for that specific discipline.27 The process of determining which topics should be tested on a PP exam, termed the Professional Activities and Knowledge Study ("PAKS") process, is a standard practice used to determine the specific topics to be tested on a PP exam. As part of the PAKS process, a consulting psychometrician28 employed by Pearson Vue; 15 to 20 engineers who are licensed in another engineering discipline; and subject matter experts who may teach a particular engineering discipline for which the PP exam is being developed, work together to develop consensus regarding the specific topics that engineers having four years of experience practicing in that discipline need to know in order to safely practice engineering in a manner that protects the public health and safety.29 26 For a PP exam to be developed for a new engineering discipline, at least ten NCEES- member state engineering boards must request that such exam be developed, and at least one ABET-accredited program in that specific discipline must exist. 27 Over the years of development and administration of the PP exam, hundreds of licensed engineers have provided input regarding the topics that should be, and are, tested in each discipline and the relative weight given to each topic on the PP exam. 28 Pearson Vue's psychometricians who develop, score, and evaluate NCEES's exams have Ph.D. degrees in psychometrics or statistical analysis. 29 Subject matter experts selected to develop the PP exam questions are chosen based on consideration of the type of practice, such as governmental and private practice; gender; ethnicity; length of time of licensure as a P.E.; and geographic considerations. All subject matter experts must be licensed as a P.E. by a state engineering licensure board in order to The consulting psychometrician builds a questionnaire that lists the specific topics identified by the PAKS committee, and distributes an online survey to engineers who practice in the discipline for which the PP exam is being developed. The survey seeks input regarding the relative importance of each specific topic for purposes of testing to demonstrate minimal competence in the discipline. Based on the survey responses from engineers practicing in the discipline, exam specifications are developed. The exam specifications identify each specific topic to be tested on the PP exam, and the number or percentage of exam questions that will address each specific topic within that discipline. The exam specifications must be approved by an oversight committee. Once the exam specifications have been approved, the subject matter experts for that specific engineering discipline for which the PP exam is being developed prepare the PP exam questions—also termed "items"—and review and evaluate them for clarity, demographic neutrality, and other parameters, so that the items will reliably and validly test engineering ability. In computer-based multiple-choice PP exams, the questions are prepared such that for each question, there is only one correct answer and three other plausible, but incorrect, alternative choices. The individual exam questions are reviewed numerous times by the subject matter experts before they are moved into an exam question bank for use on the PP exam. Once the exam questions have been developed and banked for use on a PP exam, a standard-setting committee, consisting of ten to 15 licensed engineers having diverse backgrounds, experience, and demographic features, reviews the exam to determine the minimum passing score—or "cut score"—on the exam. The cut score equates to the ability level of an engineer serve in this capacity. As noted above, over the years of development and administration of the PP exam, hundreds of engineers have provided input in developing each PP exam. having four years of experience who is minimally competent to practice engineering at a level that protects the public health and safety. This method of using subject matter experts to examine the content of each exam question and predict how many minimally-qualified examinees would answer each question correctly is termed the "Modified Angoff Method." The standard-setting committee then takes the exam, and the psychometrician analyzes the data from the committee's exam sitting. Using this data and analysis, the standard-setting committee then reviews, and reaches consensus, regarding each question, for purposes of determining the proportion of minimally competent engineers who would answer that specific question correctly. Based on the information generated by this process, the psychometrician develops the "panel recommended passing score," with a statistical margin of error. The psychometrician presents this recommended passing score to a committee of five persons, consisting of two state licensing board members and three subject matter experts who observed the exam development process. Based on the psychometrician's recommendation, the committee makes the final decision regarding the minimum passing score for the exam. Each PP exam question is developed and evaluated using the process described above, and is placed in bank for use on a PP exam. The psychometrician uses item response theory to "calibrate"—i.e., determine the relative difficulty level of—each exam question.30 An exam question is not banked for use on future sittings of the exam unless it has had at least 200 responses on a previous exam, so that statistics for each item's performance can be generated for purposes of item calibration. 30 Item response theory is one of many psychometric methods, or tools, used to weight exam questions for purposes of creating different forms of exams having the same level of difficulty. Depending on the specific discipline, a question bank for a PP exam may consist of many thousands of questions.31 Using item response theory, the psychometrician converts the passing score to create a scale from -5 to +5, which will equate to the examinee's ability level as measured by the exam. Once the passing score for the PP exam has been determined, different PP exam "forms" are created for administration in different exam sittings. Exam "forms" are essentially different versions of the PP exam that consist of different individual questions of the same difficulty level, as determined using item response theory, for each specific topic on the exam. Thus, if a PP exam was administered, for example, in April and October, the different exam administrations would consist of different forms—meaning that the exam would consist of different questions, but the questions would be of the same difficulty level for a specific topic tested on the exam.32 Additionally, because exam item difficulty has been calibrated using item response theory, different forms of a PP exam can be given during the same exam administration at different locations.33 Importantly, because the difficulty of the exam items has been calibrated using item response theory, the different exam forms are statistically equivalent in difficulty. 31 The only items that will be used on the graded portion of the PP exam are questions that have known statistics such that they have been calibrated for difficulty. However, there may be other "pretest" questions on the exam strictly for purposes of gathering statistics regarding performance on the questions for potential inclusion in the exam item bank; these "pretest" exam questions are not graded for purposes of determining the examinee's score on the exam. 32 As Miller explained, for an administration of an exam at different locations at the same time, the form administered at a particular location consists of different questions than the form administered at another location; however, the exam forms are equivalent in terms of the number of questions addressing a particular topic and the difficulty of the items addressing that topic. 33 Using the "linear-on-the-fly" ("LOFT") method to generate different forms of the exam also helps ensures exam security, since persons sitting near each other during an administration of the exam will not have the same exam form. As a result of using these processes, including the Modified Angoff Method, and applying item response theory to calibrate the exam items for purposes of constructing different, but statistically equivalent, forms of the PP exam, examinees are not graded on how they perform against each other, but instead are graded against the cut score set for the exam. To ensure that different forms of the exam are statistically equivalent in difficulty, Pearson Vue uses the LOFT method,34 which employs an algorithm to ensure that, across all of the exam forms, all examinees get the same number of questions of the same level of difficulty on the same topics. The algorithm randomly generates, or assembles from banked exam questions, different exam forms based on the exam specifications (i.e., the specific topics tested and relative weight/number of exam questions for that topic) and the difficulty level of the exam questions, such that the different exam forms generated by LOFT are statistically equivalent to each other. Using item response theory to calibrate specific exam question difficulty based on the statistical probability of being answered correctly enables examinees taking different, but statistically equivalent, forms of the exam to be compared to the passing standard for purposes of determining whether they pass the exam. Thus, examinees are compared to an ability level—here, minimal competence—rather than to each other.35 This method ensures that all examinees take an exam of equivalent difficulty, which, in turn, helps ensure the fairness of the exam. 34 For the engineering disciplines having too small a group of examinees to employ item response theory or LOFT to generate different exam forms, each examinee takes the same exam instead of taking different forms of the exam, and the exam typically is offered only on one day, rather than multiple days, per year. 35 By way of example, Miller explained that if two examinees each answer five questions on the same topic on the exam, and one examinee answers four easier questions correctly and the other examinee answers two comparatively more difficult questions correctly, the examinee answering the two more difficult questions correctly may have a higher ability level on that particular topic, due to the comparative difficulty of the questions that examinee answered correctly. Once a PP exam is administered, Pearson Vue scores the examination and sends NCEES the information regarding whether each examinee has passed or failed the exam. Pearson Vue also provides each examinee the information regarding his or her performance on the exam compared to the minimum competence standard. The examinee's performance is expressed as a scaled score, for each specific topic tested on the exam, and for the entire exam. Specifically, using psychometric statistical methods, the ability level of the examinee is expressed as "theta," and is placed on a scale of 0 to 15 for each of the specific topics tested. The examinee's overall theta across all specific topics tested is then compared to the "minimal competence" passing standard, which is also expressed as a scaled score using the same 0 to 15 scale. After an exam is taken and scored, the consulting psychometrician analyzes this data, called "response data," for each exam question, for each examinee, to calibrate the items for purposes of determining whether a particular question performs well in discriminating ability level of the examinees. The psychometrician may recommend that an exam question be "retired" because it is not performing as a good discriminator of ability level. Examples are where an exam question is too difficult or too easy, such that it does not discriminate well in determining ability level; where an item takes too long to answer or is ambiguous; where an item has been "overexposed" by having become publicized such that future examinees have access to the question and scores on the question become high; or where an exam contains "bad pair" items, such that the answer to one item may suggest, or lead to, the answer on another similar item. Having a psychometrician involved in tracking and analyzing exam data enables such circumstances and situations—which may influence the scores on a test item for reasons not related to the examinee's ability—to be identified and corrected. Returning to a free response, paper-and-pencil exam format for the P.E. licensing exam would provide a far less objective, fair, and accurate measure for determining minimal competence for purposes of being licensed to practice engineering. It also would negatively affect the ability of licensed engineers to become licensed in other states. Due to the use of psychometric methods in developing and scoring, the PP exams are very reliable across multiple administrations of the exam— to the point that all of NCEES's psychometric-based PP exams score upwards of .9 on a scale of 1.0.36 Psychometric methods, including item response theory, are used in developing, administering, and scoring many different types of high-stakes professional and academic examinations, including medical school admissions examinations, and nursing, medical examiner, internal auditor, and architecture licensure examinations. NCEES audits approximately one-third of the exams administered by Pearson Vue on an annual basis, to independently evaluate the accuracy of the psychometric services provided by Pearson Vue, and to ensure that the exams have been created pursuant to NCEES's guidelines, procedures, and requirements. NCEES also retains independent psychometricians to review Pearson Vue's exam-related reports and analyses, to ensure that Pearson Vue is following standard psychometric rules of good practice. In sum, the use of objective psychometric methods, including the methods discussed above, to develop, score, and evaluate the PP exam ensures that minimal competence, for purposes of licensure as a P.E., is accurately measured by the exam. Role of the PP Exam in Licensure of PEs in Florida As discussed above, to be licensed as a P.E. in Florida, an applicant must have a college degree from an ABET-accredited institution, four years of 36 Test reliability refers to the degree of consistency with which a test measures a particular subject matter across different administrations of the test. A test has a high reliability score if it consistently produces similar results under consistent conditions. A 1.0 reliability score reflects perfect consistency in results across different administrations of a test. An acceptable reliability score target for high-stakes tests is .7 or higher. active experience in engineering practice, and have passed the FE and PP exams. Thus, the P.E. exam is a vital component of determining that an engineer licensed as a P.E. to practice in Florida is able to practice at a competence level that protects the public health and safety. Unlike the education and experience requirements for licensure— both of which may entail a great deal of variability in quality and breadth across applicants—the PP exam constitutes an objective, consistent tool37 to measure an applicant's level of competence for purposes of determining whether the applicant possesses the minimal competence needed to practice engineering in a manner that protects the public health and safety. As such, the PP exam constitutes a uniform measure of minimal competency for purposes of licensure as a P.E. in Florida. As discussed above, this does not mean that a person who engages in engineering work but has not passed the PP exam is incompetent; it merely means that he or she has not demonstrated minimal competency on this required objective measure of competency for licensure purposes. As discussed above, the PP exam is specifically designed to ensure that a licensed P.E. is competent to practice over a range of specific areas encompassed within a particular discipline. This is because a licensed P.E. is authorized to practice engineering within any discipline or area, subject to professional and ethical standards. The breadth of the PP exam thus helps ensure minimal competence to practice engineering in a manner that protects the public health and safety. 37 This consistency and uniformity is the direct result of the psychometrically-based exam development, scoring, calibration, and validation methods discussed above. The purpose of P.E. licensure is to inform and protect the public, which is entitled to rely on such licensure as indicating that the licensee is competent to practice engineering.38 Administration of the PP Exam During the Covid-19 Pandemic As discussed above, Petitioner has alleged that the Challenged Rule is arbitrary on the basis that it does not address contingencies for offering the exam if unforeseen circumstances prevent regular administration of the PP exam. Specifically, Petitioner points to the fact that Pearson Vue cancelled the April 2020 PP exam administration due to the Covid-19 pandemic. Pearson Vue has rescheduled the PP exams for various times and at various locations around the country in an effort to make the PP exam available for prospective examinees during the pandemic.39 Pearson Vue also is taking substantial steps to protect persons who have applied to take the PP exams during the Covid-19 pandemic. To that point, Pearson Vue has retrofitted its testing centers to help ensure the safety of the examinees as they sit for the PP exam. Specifically, the number of examinees who will be in a testing room has been reduced; masks are required to be worn by examinees and proctors; testing stations are cleaned between each use; some additional test center locations have been added; and some states have relaxed rules to allow examinees to sit for the exam in states other than the one for which they are applying for licensure. At present, the exams are not able to be offered over the internet so that examinees are able to take the exam at a remote location. A substantial reason for this is lack of exam security, which is necessary to protect and 38 As noted above, a person does not have to be licensed as a P.E. to engage in engineering work in Florida. However, if a person wishes to hold himself or herself out to the public as a P.E., then that person must satisfy the requirement to pass the PP exam, which is an indicator of minimal competence for purposes of licensure. 39 Among other things, Petitioner alleges, in paragraph 19 of the Rule Challenge Petition, that the Challenged Rule is invalid under section 120.52(8)(d) because it does not address circumstances where an examination cannot be administered due to force majeure. As maintain the exam's integrity. Additionally, the lack of guarantee of internet service reliability and functionality for every examinee is a crucial consideration, since failed internet connections could significantly affect the fairness of the exam.
Findings Of Fact Test Anxiety at the February 1994 General Contractor's Examination The General Contractor's Examination used in the licensing of individuals as general contractors is divided into three parts, each of which may be passed separately. Credit for passing any one portion of the examination is good for a period of only one year. An applicant for a license as a General Contractor is limited in taking any part of the examination to three times per year. If an applicant fails to pass all three parts of the examination in one year, therefore, the applicant must commence the examination process anew, as if none of the portions of the examination had been passed. On February 17, 1994, William T. Davenport, sat for the Contract Administration portion of the exam for the third time within a one-year period. At each of the two examinations taken earlier, he had passed one portion. The Contract Administration portion was the only part he had not yet passed. Mr. Davenport was anxious as he waited to take the exam. He fully realized that if he did not pass the Contract Administration portion he would have to start the examination process all over again. His concern was generated not only from the perspective of delay and inconvenience attendant to having to repeat the entire exam process, but also from the perspective of cost. Repeating the process would require him to pay the full exam fee after having paid already a full examination fee for each of three examinations in the year of his attempt to successfully complete the examination requirements for licensure. Different Approved Reference Lists The Construction Industry Licensing Board approves reference materials that applicants may use during the examination and issues a list of those materials periodically. The Reference List for the February examination covered the period between November 1, 1993 and February 28, 1994. On that list was a book Mr. Davenport brought with him to the examination: Practical Mathematics, 3rd Edition, Copyright 1972. As Mr. Davenport sat nervously awaiting commencement of the exam, a proctor removed Practical Mathematics from his desk indicating that use of the book was not allowed. Mr. Davenport relates his response to the incident in this way, (Tr. 26.) The anxiety was very high at that point. My concern was try to get through the exam and, when the book was taken away from me, I panicked, to be blunt. I just totally panicked. Unlike the testing period from November 1, 1993 to February 28, 1994, Practical Mathematics was not on the Reference List for the next period of testing, from March 1, through June 30, 1994. The later list, issued January 13, had been out for over a month at the time of the exam. It is likely the proctor removed the book in mistaken reliance on the later list. As it turned out, the proctor returned the book to Mr. Davenport either shortly before the examination commenced. As he did so, the proctor commented, "Well, I don't know." (Tr. 25.) Mr. Davenport did not use the book during the exam because, "[a]t that point, I was reluctant to use the book ... I didn't want the test to be invalidated and I didn't want to be challenged." Id. Not using the book proved to be critical to whether Mr. Davenport passed the Contract Administration portion of the exam. He scored a 68. A passing grade is 70. Had Mr. Davenport answered correctly question number 3, which was worth four points, he would have received a 72, a passing grade. The question involves applying a percentage. Mr. Davenport could not remember whether in obtaining a percentage it is necessary to multiply or divide. Practical Mathematics has a chapter on percentages. The chapter teaches that multiplication is the arithmetic method to use when obtaining a percentage. But Mr. Davenport guessed that division should be used. He divided by the percentage and, therefore, chose an incorrect answer. Question number 3 on the exam is one of the questions that Mr. Davenport challenged originally: A 2-man crew has consistently worked at a labor performance standard ratio of 0.85 to 1. They are selected for a job requiring 60 (standard time) man-hours to perform. They will NOT work more than 8 hours per workday. NO work will be done on Saturdays or Sundays. There are NO holidays during the time the work will be performed. According to Builder's Guide to Accounting, if the job must be finished NO later than Friday afternoon at 5:00 p.m., what is the last day that they could be scheduled to start the job? Thursday of the previous week Friday of the previous week Monday of the same week Tuesday of the same week Respondent's. Ex. No. 2. The correct answer is "(D) Tuesday of the same week." The Department's expert witness explained that the correct answer is reached by way of an algebraic formula. The formula is: "the labor performance standard ratio = x (the unknown) divided by the standard time man hours" and then that answer is divided by 2 since the crew is a 2 man crew. Applied to the problem, the formula is: .85/1 = x/60, with x, once known, divided by 2. Using the formula, the calculation goes as follows: .85/1 = x/60; multiplying both sides of the equation by 60, .85(60) = x; carrying out the arithmetic calculation, x equals 51; 51/2 = 25.5. It takes 25.5 hours, therefore, for the crew to complete the job. If, as the problem states, the job must be completed by Friday at 5 p.m. and the crew works 8 hours a day, then it will take the crew 3 full days and 1.5 additional hours to complete the job. Working backward from Friday, the crew will work 8 hours on Friday, 8 hours on Thursday, and 8 hours on Wednesday for a total of 24 hours. The crew must start on Tuesday of the same week to work the additional 1.5 hours required to complete the job. One does not need to use algebra, however, to solve the problem. One can simply obtain the number of actual hours needed to complete the job by applying 85 percent (the crew's labor performance standard) to the number of standard time manhours called for by the job, in this case, 60. Here is where Mr. Davenport needed Practical Mathematics. Not knowing whether to obtain the actual hours by multiplying .85 times 60 or dividing .85 into 60, he guessed, in error, division. Through the use of division, it appears incorrectly that the number of manhours needed is 70.588. Divided by two, to take into account that there are two members of the crew, it would take the crew 35.294 hours. If it took the crew 35 hours and a fraction to complete the job on time, the crew would need to start on Monday of the same week. "(C) Monday of the same week," is the answer chosen by Mr. Davenport. Other Distractions and the Site of the Exam The examination was conducted in the Tallahassee office of the National Assessment Institute (NAI). The NAI was under contract to the Department as the vendor to conduct the exam. At the time of the examination, the exam site was a room approximately forty feet by thirteen feet four inches in size. It contained nine tables, each 18 inches by eight feet. The tables were spaced 35 inches from each other. At each table were two straight-backed chairs. There were seven candidates present for the examination. The first and last tables were unoccupied. Each of the seven candidates were allotted 34 square feet of floor space to be occupied by the candidate, the chair, the table and materials used in the exam. On the east wall of the exam room were three plate glass fixed windows. Two of the windows are 48 inches long by 36 inches tall and one is 36 inches long by 31 inches tall. The glass is one-eighth inch thick non- commercial grade. The windows are acceptable under NAI guidelines. On the other side of the windows is a workroom that measured 15.25 by 17.5 feet. Through these windows the examination supervisor seated in the work room can monitor the performance of the proctors in the exam room and see the candidates as they take the exam. The candidates, seated to the left of the workroom, do not face the workroom. Rather, their right side is exposed to the workroom. If the tables are numbered 1 through 9 on Respondent's Ex. No. 1, beginning with 1 on the side of the room marked on the exhibit as "N" or north, candidates who were seated at tables 4 through 8 were directly exposed to the workroom windows. Mr. Davenport was seated at one of the tables exposed to the workroom windows, most likely table 3, 4 or 5, that is, one of the center 3 tables. During the exam, he could see employees through the windows moving in the workroom and hear noise from the workroom. There were four employees who were present at one time or another in the workroom. Three of these employees were also engaged in proctoring the examination. In addition to the visual diversion posed by the four employees in the workroom, Mr. Davenport could hear sounds emanating from the room. The doors to the workroom, open so that the exam supervisor seated in the workroom could hear what occurred in the exam room, also allowed sounds from the workroom into the exam room. The source of the sound was the printer working, the four workers conversing from time to time and other noises associated with an office work environment. Sight of the employees and noise from the workroom prevented petitioner from fully concentrating on the exam. Mr. Davenport was also distracted by the activities of the proctors while in the exam room. During the four hours he sat for the exam, three of the four employees he observed in the workroom were also acting as proctors. They left the workroom in a rotation in order to spell each other. During their shifts as proctors, the three monitored the exam room. Mr. Davenport felt distracted by the coming and goings of the three as they rotated in and out of the room. Although there was a table designated for the proctors at a corner outside the workroom across from table 3, they rarely sat there. They sat at one of the empty tables or walked beside the seated candidates, all the time carrying out the function of a proctor: observing the candidates during examination. The FCILB Examination Administration Manual, applicable to the February General Contractor's exam, details the responsibilities of proctors in sixteen separate counts. No. 13 reads: Proctors observe at all times and move quietly about the room. Proctors do not disturb or distract candidates during the examination. If speaking is necessary, a proctor needs to be quiet and brief as possible. Proctors avoid asking candidates to move chairs to get around them, standing too close or directly behind candidates, or rustling papers and talking to other proctors in the vicinity. Petitioner's Ex. No. 2, FCILB Examination Administration Manual, p. 2-5. Movement of the proctors was necessary during the exam because of its open-book format. It is incumbent on the proctors of an open-book exam to insure that candidates do not copy questions form the examination into their reference materials. Other Candidates Reactions to the NAI's Tallahassee Office Among the three proctors the day of the exam was Ms. Jean Love. Ms. Love is also the Office Manager of the NAI's Tallahassee Office. She has worked for NAI for over two years. Before that she worked for eight and one-half years with the Department in examination services, during which she administered exams, including acting as a proctor for exams. In addition to the daily operations of the office, she oversees the administration of examinations, a function she fulfilled at the February General Contractor's exam this year. Ms. Love did not see any unusual or distracting activities on the part of the other two proctors and did not undertake any activities, in her opinion, that would have violated any of the responsibilities of proctors, including those quoted, above, from the FCILB Examination Administration Manual. The activities during the exam in the workroom, undertaken under Ms. Love's supervision, were normal activities undertaken every day at the NAI Tallahassee Office during and outside of times of examinations. Aside from typical office activities, such as conducting telephone conversations, scheduling candidates for tests, and doing paperwork that included hand-folding documentation, there was no unusual activity the day of the exam. The only event in the workroom that contributed at all to the sound of normally quiet office activity was the validation of a single candidate's check. No complaints about noise in the workroom during the February General Contractor's Exam were registered with the NAI Tallahassee Office. Nor did any of the candidates that day complain about the activities of the proctors. Ms. Love did not learn of Mr. Davenport's complaint until after he filed his challenge to the examination questions. While a proctor may have from time to time stood near Mr. Davenport as he took the exam, none of the proctors hovered over him or, in Ms. Love's opinion, did anything that would distract the average candidate. No complaint during the examination was made by Mr. Davenport. He did not complain about inability to concentrate on the exam until after he received the exam results. During Ms. Love's two years at the NAI Tallahassee office, no candidate, prior to Mr. Davenport, had ever complained about the testing environment for any reason. The comments she has received from candidates following exams have been solely complimentary. Over the last two years, the office has administered between 15 and 20 tests per month. Complimentary comments are made, on average, by one candidate per test. In the last two years the office has received, at a minimum, well in excess of 350 compliments on the testing environment from candidates. In contrast, Mr. Davenport's complaint stands alone as the only complaint about the office testing environment in the last two years at the NAI's Tallahassee Office.
Recommendation It is, accordingly, RECOMMENDED, in the alternative: That petitioner's request for reexamination or a passing grade on the "Contract Administration, Division I" portion of the General Contractor's examination administered in February 1994 be DENIED; or, in the alternative, If the Construction Industry Licensing Board is willing to overlook the petitioner's failure to challenge his grade specifically on the appropriation of the book before the examination in the petition for formal hearing and the Department's legitimate objection to the presentation of evidence on the issue, that petitioner be allowed to sit for reexamination and, if he passes the Contract Administration portion of the exam, be credited with passing the other two portions of the exam as well as if all three portions had been passed in one year. DONE and ENTERED this 1st day of December, 1994. DAVID M. MALONEY Hearing Officer Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-1550 (904) 488-9675 Filed with the Clerk of the Division of Administrative Hearings this 1st day of December, 1994. APPENDIX Petitioner's proposed findings of fact Nos. 3, 4, 5, 6, 9, 10, 12, and 13 are adopted, in substance, insofar as material. With respect to petitioner's proposed finding of fact No. 1, the first and last sentences are adopted. The remainder of the proposed finding, and in particular the reference to Linda Chaffin, test proctor, is rejected because it is either not supported by the evidence or argumentative in nature rather than factual. Petitioner's proposed finding of fact No. 2 is rejected as against the weight of the evidence. Petitioner's proposed finding of fact Nos. 7 and 8 are rejected as unsupported by the evidence. With respect to petitioner's proposed finding of fact No. 11, the reference to Linda Chaffin is rejected. Ms. Chaffin was not identified by the evidence as the proctor who removed the book from petitioner prior to the exam. Respondent's proposed findings of fact are adopted, in substance, insofar as material. COPIES FURNISHED: William T. Davenport 336 14th Avenue, North Jacksonville, FL 32250 William W. Woodyard Assistant General Counsel D B P R 1940 North Monroe Street Tallahassee, FL 32399-0750 Richard Hickok, Executive Director C I L B 7960 Arlington Expy., Ste. 300 Jacksonville, FL 32311-7467 Jack McRay General Counsel D B P R 1940 North Monroe Street Tallahassee, FL 32399-0750
Findings Of Fact In April, 1988, Petitioner sat for the examination given by Respondent to become certified in Florida as a Professional Engineer. Petitioner received a failing grade on the examination. Petitioner received a score of 46 where a score of 48 was necessary to pass the examination. Following notification that he had failed the examination, Petitioner filed a timely challenge to question 275 of the examination, contending that he had been given inadequate credit for his answer. A perfect answer to question 275 was worth 10 points. When Petitioner's answer to question 275 was first graded, Petitioner was awarded a score of 2 points. At Petitioner's request, his answer to question 275 was reevaluated. As a result of the reevaluation, Petitioner was awarded an additional two points for his answer to question 275, so that the total points awarded Petitioner for his answer to question 275 was 4 points of the possible 10 points. Petitioner contends that he should be awarded at least six points for his answer to question 275. The examination questions were prepared by the National Council of Engineering Examiners, which prepares examination questions for a number of states, including the State of Florida. Question 275 required the applicant to answer the question by assuming certain data and by applying a certain formula. The question required the applicant to give the answer and to show how he arrived at the answer. The final answer to the question given by Petitioner was the correct answer to the question. However, in coming to his answer, Petitioner did not use the formula required by the question and he did not properly utilize the information given by the question. The answer given by Petitioner to question 275 of the examination was only partially correct. The score Petitioner received for his partially correct answer was not arbitrarily or capriciously awarded.
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is: RECOMMENDED that Respondent, State of Florida, Department of Professional Regulation, enter a final order which denies Petitioner's challenge to question 275 of the examination. It is further recommended that the exhibits filed in this proceeding be sealed. DONE and ENTERED this 14th day of September, 1989, in Tallahassee, Leon County, Florida. CLAUDE B. ARRINGTON Hearing Officer Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, FL 32399-1550 (904) 488-9675 Filed with the Clerk of the Division of Administrative Hearings this 14th day of September, 1989. APPENDIX TO RECOMMENDED ORDER IN CASE NO. 89-3254 The rulings on the proposed findings submitted on behalf of Petitioner in his letter filed September 5, 1989, are as follows: The proposed findings contained in the first paragraph of the letter are rejected as being unsupported by the record and as being argument. The proposed findings contained in the second paragraph of the letter are rejected as being contrary to the evidence presented at the formal hearing. The proposed findings contained in the third and fourth paragraphs are rejected as being argument The rulings on the proposed findings contained in Respondent's Proposed recommended order are as follows: The proposed findings contained in the first paragraph are accepted in substance. See Paragraphs 1 and 2 of the recommended order. The proposed findings contained in the second paragraph are rejected as being contrary to the evidence. See Paragraph 2 of the recommended order. The proposed findings contained in the third paragraph are accepted in substance. See Paragraph 3 of the recommended order. The proposed findings contained in the fourth paragraph are accepted in substance. See Paragraph 4 of the recommended order. The proposed findings contained in the fifth paragraph are rejected as being subordinate to the conclusions reached. The proposed findings contained in the sixth paragraph are rejected as being subordinate to the conclusions reached and as being the recitation of testimony. COPIES FURNISHED: Francisco A. Lee 3885 Edgar Avenue Boynton Beach, Florida 33436 E. Harper Field, Deputy General Counsel Department of Professional Regulation 1940 North Monroe Street Tallahassee, Florida 32399-0792 Kenneth E. Easley, - General Counsel Department of Professional Regulation 1940 North Monroe Street Tallahassee, Florida 32399-0792 Allen R. Smith, Jr., Executive Director Department of Professional Regulation Board of Professional Engineers 1940 North Monroe Street Tallahassee, Florida 32399-0792
The Issue The issues herein concern the question of the Petitioner's entitlement to retake portions of the examination of the Board of Optometry which he failed in July, 1982. Petitioner proposes, in his request for relief, to stand those elements of the examination in lieu of the current style of examination which is a national examination. The prior examination had a local emphasis. In addition, Petitioner seeks additional time between re-examination segments to allow nourishment and exercise to address symptoms of an alleged handicap.
Findings Of Fact Petitioner is a licensed optometrist practicing in the State of Pennsylvania. He graduated from Pennsylvania State College of Optometry in 1959 and began his practice of optometry in that state in 1962. In 1979, Petitioner was stricken with Stevens-Johnson Syndrome, a form of autoimmune reaction. As a result of this circumstance, Petitioner was hospitalized for a period of several months and suffered loss of skin and hair and temporary blindness. In the course of this illness, Petitioner suffered extreme high temperatures and lost 65 pounds in the period of 18 days. As a result of his affliction, Petitioner was advised to locate in a climate which was warmer than that of Pennsylvania, in view of the fact that his condition did not allow him to tolerate cold temperatures. Petitioner is also an insulin variable diabetic and that condition was brought about by the treatment for Stevens-Johnson Syndrome. The diabetic condition, in the face of the Stevens- Johnson Syndrome, is difficult to treat because utilization of the compounds which control diabetes cause complications with Stevens-Johnson Syndrome. Subsequent to the time that the problem occurred with the Stevens-Johnson Syndrome, Petitioner suffered a miocardial infarct. These events related to Petitioner's health have made it difficult for him to be involved in sustained activities which require use of his vision, such as academic studies. In response to his health circumstance, Petitioner decided to seek a license to practice optometry in the State of Florida. Initially, in correspondence dated June 2, 1980, a copy of which may be seen as Petitioner's Exhibit No. 1 admitted into evidence, Petitioner requested licensure based upon reciprocity between the states of Pennsylvania and Florida. This correspondence was addressed to Mildred Gardner, who is the Executive Director of the Board of Optometry. In that correspondence, Petitioner also indicated the problems experienced with Stevens-Johnson Syndrome, his diabetic condition, and the miocardial infarct. Petitioner wrote to Ms. Gardner again on July 10, 1980. A copy of that correspondence may be found as Petitioner's Exhibit 2 admitted into evidence. In this correspondence, he alludes to a letter from Ms. Gardner of June 10, 1980, and his concern that he not be required to undergo the study necessary to stand an examination for licensure in the State of Florida. Petitioner also expressed his desire to communicate with members of the Optometric Board in the State of Florida. Dr. Edward K. Walker, a member of the Florida State Board of Optometry, advised the Petitioner that he would have to undergo the examination for licensure and would not be entitled to licensure by endorsement or reciprocity. In response to this indication, Petitioner wrote to Dr. Walker on August 13, 1981. A copy of that correspondence may be found as Petitioner's Exhibit No. 3 admitted into evidence. In this letter, Petitioner explains his concern about the ability to undertake the necessary studies to prepare to stand the license examination. Particular emphasis is placed upon the problem with diabetes which hinders his recall function and the fact that variance in the blood sugar level during the course of his day will determine the outcome of the quality of Petitioner's performance in an examination. In the letter, Petitioner also alludes to his problems related to the Stevens-Johnson Syndrome which makes it difficult to sustain "near work". Finally, Petitioner indicates that his cardiologist had suggested that the Petitioner should not be involved with induced tension. Petitioner later told Dr. Walker, in the course of a telephone conversation, that his condition was such that he could not spend a great deal of time studying, and would not be able to stand any "long term" examination, because of his health. In this discussion, Petitioner was told he would have to stand the examination. Subsequently, Petitioner requested the necessary forms to make application for licensure. Petitioner made application for examination and was provided examination information as set forth in Petitioner's Composite Exhibit No. 4. The information was related to the July 23-25, 1982, sitting. The examination information pamphlet pointed out that the written portion would be given on July 23, 1982, and specifically set forth the various sections within that examination schedule. The examination written portion was for a period of 10 hours with an hour break for lunch. Petitioner was provided a telephone number and the name of Mr. Thomas P. Gabriele, should he need further assistance on the topic of the examination. Petitioner was not made aware, prior to standing the examination, of the existence of Rule 21-11.8, Florida Administrative Code, which indicates that examinees who have certifiable handicaps and need special consideration or accommodation must make necessary arrangements with the Department of Professional Regulation before the examination date. In the course of the preparation for the examination, Petitioner sustained a problem with a vitreous detachment in one of his eyes. In effect, the retina became detached. Consequently, Petitioner would be able to read for a period of 15 minutes in his study, but then had to cease the activity. Petitioner's diabetic condition also was fluctuating during the course of his preparation for the examination and affected his vision. Petitioner also would become "giddy" during this period due to the diabetic condition, and had difficulty concentrating. He also suffered palpitations and heart pain during the sequence of preparation. Notwithstanding the problems encountered in preparing for the examination, and the concern which Petitioner had about the rigorous schedule on the day the written portion of the licensure examination would be given, Petitioner did not notify the State of Florida, Department of Professional Regulation, Office of Examination Services, about problems either through contact with Mr. Gabriele or other persons within the testing agency. Had Petitioner made contact and requested special consideration for his diabetic condition and related matters, he would have been allowed to bring food into the room, to take breaks, to provide himself with insulin, or other special Opportunities. The license examination of July 23, 1982, was con- ducted by an employee of the Department of Professional Regulation who was experienced in the administration of examinations and with the assistance of proctors who were trained in the proper methods of examination administration. On July 23, 1982, the date of the written examination, Petitioner managed to perform in an acceptable manner in the initial parts of the examination; however, at approximately 11:30 a.m., Petitioner began to experience difficulty. Those difficulties were related to problems with the insulin which he was receiving, to the extent that the words on the examination documents became blurred and his hands began to shake. Nonetheless, he was able to complete that aspect of the written examination because of the ability to work from an open (book). That particular phase of the examination did not require the utilization of recall. In summary, Petitioner was successful as a candidate on those portions of the examination given on the morning of July 23, 1982. At lunchtime, Petitioner did not find the available food in the restaurant in the examination facility acceptable, in view of his diabetic condition and only was able to eat toast and a salad to sustain him. During the lunch break, Petitioner also ate dried fruit, which he had provided for himself, and took a tranquilizer. After taking his lunch, Petitioner still felt ill. He began to notice that his lip gave him a tingling sensation and had difficulty sustaining his efforts at reading. He realized that he was going to have problems with the afternoon portion of the examination, in that in addition to other symptoms described, he began to have palpitations related to his heart problem. In the afternoon testing session, Petitioner's lip started to swell beyond dimensions noted at lunchtime. Respondent felt "giddy" and subject matter which he felt was relatively easy to answer in the examination was difficult to comprehend. When he attempted to obtain food from his coat pocket, he was advised that he could not take things out of his pocket. This admonition was given based upon a concern that candidates for licensure not cheat in the course of the examination process. Petitioner did not adequately explain the reason for his attempt to retrieve the food from his pocket, nor did he at any time during the course of the license examination on the date in question, explain to examination officials that he was experiencing physical problems. Petitioner did not pass those portions of the written examination administered in the afternoon of July 23, 1982, although he attempted to overcome his physical problem related to concentration, associated physical manifestations, to include vision and induced euphoria. Those subjects were pharmacology and theory and practice of optometry. (The Florida Board of Optometry now utilizes a national optometry examination in substitution for the examination in pharmacology and the theory and practice of optometry which was given in the afternoon of July 23, 1982.)
The Issue The issue for consideration in this case is whether Petitioner should be awarded additional credit for his answer to question number 290, and thereby be given a passing grade on the Professional Engineer examination administered on October 25, 1996, in Orlando.
Findings Of Fact At all times pertinent to the issues herein, the Board of Professional Engineers (Board) was the state agency responsible for the examination and licensing of professional engineers in Florida. With the cooperation and assistance of the National Council of Examiners for Engineering and Surveying (NCEES), the Board conducts periodic examinations to test the qualifications of candidates for certification as professional engineers in this state. Such an examination was conducted in Orlando, Florida on October 25, 1996. Petitioner was a candidate at that examination. A minimum score for passing was 70. Petitioner received an overall score of 69. One of the questions posed to the candidates at that examination was question number 290, dealing with the design of a control system, which required the candidate to determine values for two parameters in such a fashion that the closed loop specifications stated as, "with K =20 the unit step response be a damped oscillation with a 10% overshoot and with a damped natural frequency of 15 rad/s" were met. In the answer to this question, the engineer has to arrive at parameters to give the desired step response within the stated percentages. According to Dr. Antonio Arroyo, an assistant professor of electrical and computer engineering at the University of Florida and an expert in electrical engineering, this subject matter is taught in a standard undergraduate controls course which is required in engineering schools nation-wide. The question in issue is a classic controls problem. The candidate is to reduce the diagram displayed in the examination question and give a closed loop description. Given that, the solution proceeds by taking the percentage of error and using it to back- track and arrive at the requested parameters, step by step. The examination is an open book examination. Because of the many formulae used in engineering, the candidate is permitted to use printed resources to assist in the solving of the problems. This formula involved in this problem is standard. Only the parameters cited in the test problem are different. In his answer to the question Petitioner cited to the page in his reference material where the solution is to be found, and he used the appropriate formula. In doing so, he could take the numbers presented in the problem and apply them to the standard problem solution contained in the reference book he had with him. It is a "plug and chug" situation wherein the candidate inserts the problem numbers into the given formula and makes the calculations. In order to take advantage of this opportunity, however, the candidate must decide how to use the information given. In this case, the problem involved a damped frequency of 15 hz and the candidate was required to calculate an undamped frequency. The Petitioner did not show that calculation in his solution, and it appears to Dr. Arroyo he missed the fact of the difference between the two frequencies. In Petitioner's solution, he listed what he saw as the data given, and though at no place did the problem show "Omega d", Petitioner put down "Omega d" but used "Omega n". In the expert opinion of Dr. Arroyo, an engineer should, at least, check his calculations. Examiners will give credit to a candidate if the candidate shows the appropriate knowledge of the concepts involved in the problem. In the instant case, Petitioner's answer to question 290 far exceeded the allowable 10% overshoot. His answer for "a" was 0.895, whereas the correct answer was 1.099. Whereas the allowable ten percent difference was .110, Petitioner’s overshoot was .204. His answer for "b" was 11.25, whereas the correct answer was 17.3. Whereas the allowable ten percent difference here was 1.73, Petitioner’s overshoot was 6.05. To Dr. Arroyo, this shows a concept error rather than a calculation error In substance, Petitioner utilized the correct formulae, but used incorrect data, and the use of the wrong data is sufficient to indicate his ignorance of the appropriate concepts. Petitioner’s expert, Dr. Garrett, who did not see the problem utilized in the examination and relied on information provided by Petitioner, concluded that Petitioner’s margin of error was within the 10% limitation. Here, notwithstanding the opinion to the contrary of Dr. Garrett, Petitioner's solution missed the authorized overshoot by a significant amount, far more than the allowable 10%. He should have known something was wrong when this happened and should have looked to see what he did wrong. In the opinion of Dr. Arroyo, the Petitioner did not adequately evaluate the problem consistent with acceptable engineering standards since the final product of his calculations did not meet the specifications of the problem. This is the purpose behind the professional certification process, and Petitioner should have recognized that his answer did not meet the required specifications. Petitioner received a score of six out of a possible ten for his solution to question 290. Dr. Arroyo is satisfied that the scoring plan of the NCEES for this problem is fair and he supports it. Petitioner’s expert, Dr. Garrett, a professional engineer and long-time professor of electrical engineering at the University of South Florida, disagrees. In his evaluation of the problem and the grading process used here, Dr. Garrett notes that problem 290 consisted of five parts, for each of which two points could be awarded. Petitioner correctly answered the first three parts and received a grade of six points. He missed part four, and part five was to use the results of parts three and four, with the proper equations, to determine the two answers required. Since Petitioner used the proper equations to figure his answer to part five, even though he did not get a correct answer to part four, which resulted in his numeric answer to part five being incorrect, Dr. Garrett is of the opinion that he should have received an additional two points for applying the proper formula in part five. Review of the scoring plan developed for this problem indicates that Petitioner met all the qualifications for award of six points, but he did not recognize the relationship of damped as opposed to undamped. He used incorrect data to arrive at "a" and "b" in that he did not identify the relationship between natural frequency and damped frequency. This is a basic problem of control systems which an undergraduate should be able to solve correctly. It is basic electrical engineering knowledge and not beyond that expected of an electrical engineer with a bachelor's degree in the field. Had Petitioner utilized the formula he used with the proper data, he would have been awarded credit for a correct answer even if his calculations were incorrect. Here, however, while Petitioner utilized the correct formula, he applied it to incorrect data, and it is this use of incorrect data which makes an award of a higher score inappropriate. The professional engineers’ examination is designed to test the individual's familiarity with engineering concepts and his ability to cast the problem into those concepts to solve the problem. Petitioner contends that his understanding of the concepts involved was correct and, therefore, even though he used the wrong figures, he should received credit for a correct answer or, at most, only 2 rather than 4 points should have been deducted. Though Petitioner utilized the correct formula for his solution to question 290, he applied the wrong values in the use of the formula. This indicates a lack of understanding of the concepts involved, and even though Petitioner used the proper formula, that formula came from the book he was permitted to use for the examination. He cannot be given full credit for copying the formula from the book. Had he used the correct values in his solution to the problem, he would have been given appropriate credit even if his calculations were wrong. After being notified of his unsuccessful exam results, Petitioner requested that his answer to question number 290 be resubmitted to NCEES for re-scoring, and this was done. By memorandum in response, dated July 10, 1997, the NCEES scorer concluded: The error in using undamped natural frequency for damped natural frequency in the examinee's solution is a major error. Whether the examinee did not recognize the function was in fact the undamped natural frequency, as given in the problem statement, or whether it was an oversight, it is still a major error since the outcome is significantly affected. The scorer, whose knowledge of the identity of the candidate was limited to a number only, recommended a score of "six" for Petitioner answer to this problem. There was no change from the initial scoring.
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is recommended that the Board of Professional Engineers enter a Final Order denying Petitioner additional credit for his answer to question number 290 on the principles and practice portion of the electrical engineering examination administered for the Board of Professional Engineers on October 25 and 26, 1996. DONE AND ENTERED this 17th day of December, 1997, in Tallahassee, Leon County, Florida. ARNOLD H. POLLOCK Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (904) 488-9675 SUNCOM 278-9675 Fax Filing (904) 921-6947 Filed with the Clerk of the Division of Administrative Hearings this 17th day of December, 1997. COPIES FURNISHED: Bahman Behzadi Post Office Box 290931 Tampa, Florida 33687 R. Beth Atchison, Esquire Department of Business and Professional Regulation 1940 North Monroe Street Tallahassee, Florida 32399-0750 Lynda L. Goodgame General Counsel Department of Business and Professional Regulation 1940 North Monroe Street Tallahassee, Florida 32399-0700 Angel Gonzalez Executive Director Board of Professional Engineers 1940 North Monroe Street Tallahassee, Florida 32399-0792
Findings Of Fact Petitioner herein, Kristina V. Tignor, took the Professional Engineers Examination for the State of Florida in Orlando on April 9 and 10, 1987. On July 22, 1987 she was advised by the Department of Professional Regulation's Office of Examination Services that she had failed the examination and was given a cummulative score of principles and practice of 69.1 percent. In her initial request for review and reconsideration, Petitioner objected to the points assigned to her solutions for three problems on the test, Numbers 425, 421, and 124. She contended that as a working engineer, certain criteria and assumptions must be made in approaching any engineering problem and, because the portion of the examination in issue is graded subjectively, her answered should be reconsidered and evaluated in that light. At the hearing, Petitioner contested only the grading of questions number 124 and 421, thereby accepting the grade given for question 425. With regard to Question 124, Ms. Tignor was awarded a score of 5 on her solution to this problem. The National Council of Engineering Examiners, in its Standard Scoring Plan Outline awards a "qualified" evaluation to scores from 10 down to 6 on this question. Scores from 5 to 0 are rated, "unqualified." A score of 5 indicates the applicant has failed to demonstrate adequate knowledge in one aspect of one category. Specifically, a rating of 5 in this question indicates that the examinee displayed an inadequate knowledge of weight/volume and concrete mix design. Her computations were displayed and an incomplete or erroneous solution was arrived at which gave a generally unrealistic result. Dr. Bruce A. Suprenant a civil engineer registered in four states and who teaches engineering at the University of South Florida, reviewed the question, the Petitioner's solution, the solution proposed by the examiners, and the grading scheme for this problem and found a number of illogical items in Petitioner's solution which, to him, were difficult to understand. He found several items which had no basis and which were possibly assumed. As to Part a of Petitioner's answer, a mixture of answers, (correction for moisture), which should have been in Part b, was located in Part a. As to density, the value used by Petitioner does not appear to be reasonable based on information provided in the problem. In Dr. Suprenant's opinion, there are at least three approaches to this problem. One is the water/cement ration method. Another is the weight method. The third is the absolute volume method. The water/cement ratio method would be difficult to apply here and neither Petitioner nor the examiners used it. As to the weight method, much the same problem exists. There is insufficient information provided to satisfactorily apply this method and while the examiners did not use it, Petitioner did. Petitioner's answer has a correction for moisture in the absolute volume method on the first page of the solution form at the top. The calculations by Petitioner are assumed information not known, (volume). In addition the correction for moisture in the second part of page one is included on the top of page two. It is not a part of the solution for subpart a and should not be there. Petitioner used 150 pounds per cubic foot for concrete density in her solution and this choice is not explained. Most publications utilized by engineers suggest using tables which were not provided to the examinees and it is, therefore, illogical to assume concrete density with no history for that assumption. Petitioner's answer of 5.41 cubic yards is only slightly off the suggested answer of 5.44 cubic yards but the fact that the answers are close does not justify her assumption. It might well not come so close in other cases. As to Part b of the question calling for the water/cement ratio, the corrections for moisture of fine and coarse aggregate on page one are acceptable. On the second page, a problem arises in when the correction for moisture should decrease. Petitioner got the right factor but applied it in the wrong manner. As a result, her answer to Part b of the examination question is wrong. Her answer was 4.40 as opposed to the correct answer of 4.34. This small degree of error can be attributed to the smallness of the amount in question. Were the amounts greater, the error would be greater. As to part c of the question, which deals with the cement factor in a yard of concrete, Petitioner's approach of dividing sacks of cubic yards is correct, but the cubic yard content was determined from Part a of the question, and Dr. Suprenant does not agree with how she got her solution. He therefore questions her carryover. The standard weight of a sack of concrete is 94 pounds. The individual grading Petitioner's response to Question 124 indicates she displayed inadequate knowledge and reached a solution which gives "unrealistic results." Dr. Suprenant agrees, contending that Petitioner's performance in regard to this question indicates inadequate knowledge of weight/volume relationship. She made inadequate assumptions in formulating her answer to the question. The fact that in this problem she arrived at a solution close to the correct one does not indicate that in other problems, she would achieve the same closeness using the same procedure. In his opinion, Petitioner showed some confusion regarding the basis for solving this problem and Dr. Suprenant believes that a grade of 5 as awarded by the examiner is correct. Petitioner questioned the fact that the various technical weights and volumes, such as 94 pounds in a sack of concrete, 8.33 pounds for a gallon of water, and 27 cubic feet in a cubic yard do not appear in the problem statement. This, in the opinion of Dr. Suprenant, compounds the gravity of Petitioner's deficiency. They are routine "givens" generally accepted in the practice by engineers and it would be difficult to assume that anyone familiar with the practice of engineering would use different "givens" for these specifics. Petitioner's employer, Mr. Bishop, himself a registered civil engineer in Florida since 1958, also reviewed Petitioner's solution to Question 124. He admits that on the first page of the answer sheet, Petitioner began solving the problem in an inappropriate way. Her calculations for moisture content were correct, however. On the second paged the correction factor was put in with the wrong sign and the aggregate was given the wrong factor. As a result, the answer was off. In his practice, however, the error committed by Petitioner in these regards is both minimal and acceptable. Her choice of 150 pounds per square foot is reasonable and produced a close result, and while it is true that if the project were of a greater scale, the error might be significant for a test question, as here, the error, in his opinion, is insignificant. He feels much the same way regarding the error in Part c of the examination question. While the factors used by petitioner were wrong, the process used was correct and the answer was not unreasonably incorrect for a test solution. In an examination situation, the calculations are not being done on a continuous basis, and he feels the grade of 5 awarded is unduly harsh since the error was numerical rather than operational. In his opinion, a more reasonable grade would have been a 6 or 7. Petitioner began her solution to this problem by using one similar to that used by the examiners in their publications. Shortly, however, she realized she would not get the answer she needed by doing so and abandoned her solution. She forgot to cross it out, however, and now recognizes she should have done so. She thereafter began to accomplish a series of new calculations on the first page of the answer sheet but did not necessarily utilize that data for her solution to Part a. She admits she made an error in calculation for moisture on the second page. In that calculation, she used the study manual and admits now that she should have cited the figure she used. As to Parts b and c, her use of some figures from Part a may have thrown her answer off somewhat. However, the 5 awarded her, indicating her solution was unrealistic, is, in her opinion unfair as she considers her answer to be quite realistic. The problem did not state what solution method to use and she feels her use of givens from recognized manuals such as the 150 pounds, should not be held against her. 94 pounds for a sack of cement used by the grader was also not given and her use of other accepted numbers should not, she contends, be held against her. Petitioner believes a grade of 7 would more accurately describe the quality of her answer. A 7 means that the examinee obtained an appropriate solution but chose a less than optimum approach. The solution is, therefore, awkward but nonetheless resonable. Ms. Tignor believes that while her approach may have been awkward, she achieved reasonable solution demonstrated by the fact that it was only slightly off the correct figure. Therefore, she believes a grade of 6 would be appropriate. This examination was an open book examination and Petitioner had her manuals with her. She could have easily determined the appropriate weights an "givens" from these manuals without choosing those she used. Ms. Tignor's conclusions that her results are realistic are contradicted by the Board's expert. Realistic results are, in engineering practice, not only the figure reached but also the method used in arriving at that figure. Here, though Petitioner's results are close, the approach utilized in arriving at her solution is unrealistic. Her approach showed an inadequate knowledge of weight/volume and calculations. Consequently it is found the grade is valid and was not arbitrarily assigned. According to the Standard Scoring Plan Outline, each score from 10 through 6 has an indispensable criteria that all categories must be satisfied. Since Ms. Tignor's examination response did not satisfy all categories, the best she can be given is a 5 and that award appears to be justified by the evidence presented. Question 421 was a four part drainage problem. Petitioner used as a part of her solution calculations based on a 100 year storm and this was determined by the examiners to be inappropriate. Ms. Tignor was awarded a grade of 8 and contends she was not given appropriate credit. She relates that even Mr. Smith, the Executive Director of the Board of Professional Engineers, advised her she may not have been given full credit for her answer. She was given full credit for Part a but lost two points for part c which included a calculation error to which Petitioner admits. She contends however, it was so minor, only one point should have been deducted. Were Petitioner to receive an additional one point on this question, she would pass the examination which she failed by only one point. However, this issue must be resolved on the basis of lawfully admitted evidence and Mr. Smith's comment, being unsupported hearsay evidence, cannot itself sustain the rasing of the grade. The Standard Scoring Plan Outline for this question reflects that to receive an 8, the examinee must demonstrate that all categories are satisfied, that errors are attributable to misread tables or calculating devices, and that errors would be corrected by routine checking. The results must be reasonable if not correct. For a 9, the examinee must demonstrate that all categories are satisfied; that a correct solution is arrived at but the examinee has been excessively conservative in the choice of working values; and that examinee's presentation is lacking in completeness or equations diagrams or orderly steps in solution, etc. Subqualifications for a 9 indicates that the answer is correct but that the organization of the solution is not logical. One error in calculation in any of the Parts from a to d, which does not affect the other parts of the solution, is acceptable. Mr. Kenneth Weldon, the Assistant State Drainage Engineer for the Department of Transportation, an expert in the area of drainage to which this problem relates, reviewed the question and the Petitioner's answer thereto and would award a grade of 8 to her answer. He found various numerical mathematical errors which led to the wrong solution. In addition, Petitioner made various assumptions that, though supposedly supported, were, he felt, in error through her misinterpretation. In general, none of the actual solutions she arrived at were correct. Specifically, that portion of the problem to determine the cross sectional area of the waterway for establishing normal depth flow was done incorrectly. Because the Petitioner used incorrect equations throughout the problem, the depth flow computed is high. Petitioner did no analysis to determine whether or not any of the several situations relating to flow control were pertinent. Mr. Weldon initially felt Petitioner's answer to the question merited a grade of 6. This means that the examinee knew all the proper steps but failed to interpret some of the criteria properly. He could not award her a grade of 9 which would indicate all categories were satisfied and the solution was correct, if conservative. Petitioner's solutions were incorrect. He subsequently changed his award to an 8, however, on the basis that the Petitioner's errors were attributable to a misread table or calculating device and would be corrected by routine checking. The result was reasonable, though not correct. Mr. Weldon did not like this question even though he believed it appropriate for a one-hour exam. As written, it involves establishing and making judgements beyond what someone minimally competent would be expected to do. It requires materials that are beyond what are normally available to someone taking the exam. However, Petitioner failed to make proper provision to protect herself in a case where the question is inappropriate or incomplete. If she felt something was wrong with the question, she should have clearly stated the assumption she was making to solve the problem. This was her responsibility and she failed to do so. In Mr. Weldon's opinion, Petitioner's answer might merit a grade slightly higher but not significantly higher. His reasoning is that Petitioner misinterpreted the criteria she stated for writing the problem. Her comment that the Department of Transportation uses 100 year storm criteria was incorrect even though that statement is made in outdated Department of Transportation publications. The basis for her answer is not well established or correct, or based on engineering calculations or judgement, and at best he could award no more than an 8.
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is, therefore: RECOMMENDED that a Final Order be entered affirming the score awarded to Petitioner on questions 124 and 421, respectively, of the Civil Engineering Examination administered to her in April, 1987. RECOMMENDED this 10th day of June, 1988, at Tallahassee, Florida. ARNOLD H. POLLOCK, Hearing Officer Division of Administrative Hearings The Oakland Building 2009 Apalachee Parkway Tallahassee, Florida 32399-1550 (904) 488-9675 Filed with the Clerk of the Division of Administrative Hearings this 10th day of June, 1988. APPENDIX TO RECOMMENDED ORDER, CASE NO. 87-5110 The following constitutes my specific rulings pursuant to Section 120.59(2), Florida Statutes, on all of the Proposed Findings of Fact submitted by the parties to this case. For the Petitioner None For the Respondent Accepted and incorporated herein. Accepted and incorporated herein. Accepted and incorporated except for the characterization of several assumptions as guesses. No evidence exists to support such a characterization even though they are incorrect. Accepted and incorporated herein. Accepted and incorporated herein. Accepted and incorporated herein. Accepted and incorporated herein. Accepted and incorporated herein. COPIES FURNISHED: Kristina V. Tignor, pro se 2160 North Oval Drive Sarasota, Florida 34239 H. Reynolds Sampson, Esquire Department of Professional Regulation 130 North Monroe Street Tallahassee, Florida 32399-0750 Allen R. Smith, Jr. Executive Director DPR, Board of Professional Engineers 130 North Monroe Street Tallahassee, Florida 32399-0750
The Issue The issue is whether Rules 21H-21.002(1) and 21H-21.004(1) are an invalid exercise of delegated legislative authority.
Findings Of Fact Ronnie F. Taylor, of Post Office Box 697, Cedar Key, Florida, is employed by the engineering firm of Ingley, Campbell, Moses and Associates of Gainesville, Florida, which engages in mechanical, electrical and plumbing engineering. Taylor has been with this engineering firm for four years and is currently a vice president in charge of production of electrical engineering documents. Prior to this employment, Taylor spent 14 years as an electrical engineer with the engineering firm of Reynolds, Smith and Hill of Jacksonville, Florida. When Taylor left Reynolds, Smith and Hill, he was the senior design engineer. Taylor served in the military as an electrician. Upon completing military service in 1967, Taylor entered Florida Junior College. He received an Associate of Science degree in Electrical Engineering Technology in 1970 from that institution. Following that degree, Taylor began employment with Reynolds, Smith and Hill, where his responsibilities included the design of electrical projects for commercial buildings, including writing specifications, making cost estimates and producing a finished product. Taylor has spent his entire career in electrical engineering and has no experience with other specialties of engineering. He has extensive experience in electrical engineering having designed and completed numerous large commercial projects. However, because Taylor is not a licensed professional engineer, a licensed professional engineer must oversee all projects during the course of design and completion and must sign and seal all completed work. Taylor is not a licensed professional engineer because he has failed to pass the Fundamentals of Engineering (FE) portion of the engineering examination. He has failed in fourteen attempts to pass the FE exam. Taylor did pass the Principles and Practices (P & P) portion of the exam in 1982. Licensure requirements specify that both sections must be passed prior to licensure. Taylor became qualified to take the engineering exam in 1977 pursuant to Section 471.21(1)(c), Florida Statutes (1977), which permitted an applicant to take the exam with "a specific record of 10 years or more of active practice in engineering work of a character indicating that the applicant is competent to be placed in responsible charge of such work." This so-called 10 year cycle permitted an applicant to qualify for the exam without the otherwise required 4- year college degree and 4 additional years of experience. In 1979, Section 471.013, Florida Statutes, was enacted, allowing persons in the final year of engineering school to take the FE exam to qualify as an engineer intern. This provision has been in effect since 1979. The FE exam, as required by Rule 21H-21.002(1), which is challenged here, includes questions on the subjects of mathematics, mathematical modeling of engineering systems, nucleonics and wave phenomena, chemistry, statistics, dynamics, mechanics of materials, fluid mechanics, thermodynamics/heat transfer, computer programming, electrical circuits, statics, structure of matter, engineering mechanics, electronics and electrical machinery. While Taylor scored highly on the subjects relating to electrical engineering, he had difficulty with other areas of the exam. The course work completed by Taylor in 1970 did not include some of these areas with which Taylor had difficulty. Taylor has had no course work in computer programming, thermodynamics, statistics, nucleonics and wave phenomena. The subjects tested in the FE exam are updated in order to test applicants on the most current information and knowledge of engineering fundamentals. Herbert A. Ingley is a licensed professional engineer and holds a Bachelors degree in Chemical Engineering, a Masters degree in Mechanical Engineering, and a Ph.D. in Mechanical Engineering with a minor in Environmental-Mathematics. He taught full time on the faculty of the University of Florida in Mechanical Engineering for 11 years. In his opinion, it is more difficult for applicants to pass the FE exam the further they are from their formal education and, therefore, applicants in the 10 year cycle have more difficulty passing the exam. According to Ingley, the requirement that persons such as Taylor wait 10 years before taking the FE exam is not logical. However, Ingley also opined that it is important for a professional engineer to have a fundamental knowledge of engineering and that there is a need to test the fundamental basics of engineering for each person who is going to become a licensed professional engineer. George Edward Rabb is a licensed professional engineer, having been licensed in 1965. He was grandfathered and therefore only had to pass the P & P exam. The FE exam was waived based on specific portions of statute and rule which waived the FE exam for persons with fifteen years experience. The waiver was only available to persons qualifying prior to November, 1970. According to Rabb, an engineer needs to have a working knowledge of fundamentals and to understand the general concepts of engineering. Robert D. Kersten, who has been the Dean of the Department of Engineering at the University of Florida for 20 years, has a Bachelors degree in Mathematics and Chemistry, a Masters degree in Civil Engineering, and a Ph.D. in Civil Engineering, Water Resource/Hydrologic Engineering. Dean Kersten has served in numerous capacities with both state and national professional associations involved in accreditation of engineers and served on the Board of Professional Engineers in Florida and on the National Council of Engineering Examiners. The FE exam is prepared by the National Council of Engineering Examiners and is designed to cover the fundamental areas essential to the basic practice of engineering. The FE exam tests both the common body of knowledge that is essential to practice in the profession and the ability to apply that knowledge. According to Dean Kersten the FE exam tests items which should be within an engineer's basic knowledge and which are necessary to communication between engineers in a design team approach to project design. Dean Kersten acknowledges that the FE exam is more difficult for applicants who lack a degree or who have been out of the academic area for a period of time, but opines that those factors do not excuse an applicant from mastering and retaining the basic fundamentals important to the practice. In fact, the FE exam is designed so that 70 percent of the applicants with-the 4- year college educational background pass the exam. Only 40 percent of the applicants in the 10 year cycle pass the exam.
Findings Of Fact The Petitioner, James Ilardi, was an unsuccessful candidate for the June, 1989 General Contractor's Construction Examination in the State of Florida. He is an experienced contractor and is licensed in the State of South Carolina. He is the past President of the Charleston Contractor's Association in South Carolina. He has experience with most types of building construction, including office buildings, military facilities, hospitals, factories and other large projects. He served as Chief Executive Officer for a design and construction firm for a period of ten years. The Respondent is an agency of the State of Florida charged with administering the certified general contractor's examination and with regulating the licensure and practice of construction contractors in the State of Florida. The Petitioner sat for the certified general contractor's examination in June, 1989. He has challenged the scoring of his answers to questions 2, 11, 12, 17, 19 and 33 on that examination. During the course of the hearing, he abandoned his challenges to questions 2, 17, 19 and 33. If he were accorded correct answers to either of the remaining challenged questions, numbers 11 or 12, he would have a sufficient score to obtain a passing grade of 70 on that examination. Both questions 11 and 12 used a "critical path network diagram" for use in working out the correct answer to the questions. The Petitioner criticized the diagram as being obscure, difficult to read and containing error. He maintained that it was not supported by the representations found in the reference materials recommended by the Respondent, in its "Instructions to Candidates", as being the material to use to arrive at answers to the questions. The Petitioner contends that the size of the diagram, "the multiple fonts, the difference in the intensity of the print, and the use of symbols all contribute to the obscurity and illegibility of the diagram, itself". In particular, he complains that the symbol listing includes a symbol which he did not find on the diagram. That is, the symbol for "structural steel" and "steel bar joists, which is two straight vertical parallel lines. He also complains that general practice in the construction industry, in his experience, and as indicated in the reference work "Construction Contracting", pages 325-326, one of the references listed for candidates to use in answering these questions, recommends against the use of symbols in lieu of abbreviated notations for description of activities on such a diagram. The main complaint he had concerning the use of symbols, however, was the fact that use of symbols, and having to constantly defer to the symbol legend on the exam materials, was time-consuming and was not generally accepted industry practice or procedure. He contends that the diagram contains error or is obscure and does not conform to the Respondent's recommended reference materials nor to industry standards and. is deficient in format, design and reproductive quality. Thus, he maintains that questions 11 and 12 do not adequately test the knowledge or skills necessary for licensure as a general contractor. The Petitioner acknowledged that the questions at issue had been reviewed twice by the Respondent's examination content specialist and that an "item analysis and review process" by the Respondent's expert resulted in the Respondent maintaining its position that the two questions and supporting materials were valid in fairly testing the knowledge of general contractor licensure candidates. In summary, the Petitioner contends that as to question 11, the symbol for steel bar joists, the two parallel vertical lines, does not appear on the diagram; therefore, he was unable to determine whether his answer was correct or not. As to question number 12, he maintains, in essence, that the use of symbols instead of brief abbreviated descriptions of the activities involved, accompanying the arrows in the diagram which indicate the critical path for the activity in question (paint work), render answering the question confusing and time consuming in having to constantly refer to the symbol legend and look for the symbols. He states that, in his 20 years of construction industry experience, he has not had to use symbols in working with a critical path diagram. The Petitioner did not demonstrate, however, that the use of symbols was incorrect procedure as delineated in the reference materials supplied to the candidates and which they were instructed to use in answering the questions on the examination. The Respondent produced the testimony of Mr. Olson, a Florida certified general contractor, who is also employed with the National Assessment Institute which developed this examination. Mr. Olson, however, did not, himself, have a hand in developing the examination. Mr. Olson did, however, review the Petitioner's challenges to the questions at issue and his responses, reviewed questions 11 and 12, as well as the Respondent's asserted correct answers to those questions and the methodology used in reaching those answers. Mr. Olson established that this was an "open-book" examination and the candidates were informed of and supplied all necessary reference materials to answer these two questions. The only optional consideration was that candidates could have used a calculator to speed up their calculations and were informed that it was permissible to use a calculator. Question 11 required candidates to calculate the total time necessary to install structural steel and steel bar joists in interpreting the activity network represented by the diagram in question. They were asked to calculate whether the installation was ahead of schedule or behind schedule and by how much. Mr. Olson established that the correct response was "C", which is two days behind schedule. Mr. Olson demonstrated that it was quite possible for a candidate to make this calculation and track this in formation on the diagram provided the candidates, through reading the path with the symbols, which alphabetically represent the activity, and which are numerical in representing the time in days. He established that this is very typical of the construction industry, related to the preparation, reading and interpreting of blueprints. A tremendous amount of symbols and legends are typically used in preparing and interpreting blueprints. Mr. Olson established that the pertinent number, 85 days, could be calculated for installation of structural steel and steel bar joists, based upon the information supplied to the candidates. By using the diagram and the information supplied with the question, the candidate can calculate that the actual number of days that were taken for the job was 87 days and therefore, that the project, at that point, was two days behind schedule. Mr. Olson performed this calculation by using the actual diagram the Petitioner used and reference information the Petitioner was given to use in answering the actual examination question at issue. He also established that the two parallel lines representing steel bar joists and structural steel on the diagram, and in the symbol legend supplied with the diagram, were indicated on the diagram supplied to Mr. Ilardi at the examination. Mr. Olson also established that the reference quoted for question number 11 was walkers Building Estimator's Reference Book, which, indeed, listed the type of activity network depicted in the diagram used by candidates for question number 11 and 12. Mr. Olson also established that question number 12 requires a candidate to work through an activity network diagram to find the amount of days necessary from the beginning of a project to the time the painting activity begins. He established that the answer could be obtained without the use of any other reference materials other than the information depicted on the diagram, itself, associated with the question. He established that the only correct answer from that information on the diagram could be "D" or 153 days. The Petitioner did not establish that his answer to question number 12, nor to question number 11 for that matter, was a correct answer and did not establish that there was any misleading quality or ambiguity in the wording of the questions and the associated information which would mislead a candidate into calculating the wrong answers or that there was erroneous information depicted in the reference materials or the diagram which would result in the candidate being misled into giving a wrong answer to questions 11 and 12.. Mr. Ilardi challenged the examination as to the testing environment, as that relates to the ambient light level in the examination room and to the acoustic qualities of the room. He also asserted that the test was not standardized throughout the State and was biased due to age, because of the perceived hearing and vision difficulties which he believed were caused by the acoustics in the examination room and the light available. Other than stating his opinions in this regard, he produced no testimony or evidence concerning these alleged qualities of the testing environment. It was demonstrated by she Respondent that, indeed, the test is standardized throughout the State and is the one given to all candidates in Florida, regardless of the test location.
Recommendation Having considered the foregoing Findings of Fact, Conclusions of Law, the evidence of record, and the candor and demeanor of the witnesses, it is therefore RECOMMENDED that a Final Order be entered denying Petitioner's request to receive a passing grade on the certified general contractor's licensure examination. DONE and ENTERED this 4th day of May, 1990, in Tallahassee, Leon County, Florida. P. MICHAEL RUFF Hearing Officer Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-1550 (904)488-9675 Filed with the Clerk of the Division of Administrative Hearings this 4th day of May, 1990. APPENDIX TO RECOMMENDED ORDER CASE NO. 89-3784 Petitioner's Proposed Findings of Fact Rejected. The Petitioner was not qualified as an expert witness. Other than that, this finding is accepted. Rejected, as not in accord with the preponderant weight of the evidence. Rejected, as not supported by the preponderant weight of the evidence. 4.A.-4.C. Accepted. 4.D. Rejected, as not in accordance with the preponderant weight of the evidence. Accepted, but not itself materially dispositive. Rejected, as not in accordance with the preponderant weight of the evidence. Rejected, as not in accordance with the preponderant weight of the evidence. Rejected, as not in accordance with the preponderant weight of the evidence and as subordinate to the Hearing Officer's Findings of Fact on this subject matter. Rejected, as not constituting a finding of fact, but rather a quotation from the transcript of the proceedings. Rejected, as not in accordance with the preponderant weight of the evidence and as subordinate to the Hearing Officer's Findings of Fact on this subject matter. Rejected, as immaterial and not probative of the issues of whether the questions were ambiguous or misleading or whether the Petitioner's answers were correct. Rejected, as not in accordance with the preponderant weight of the evidence. Rejected, as not in accordance with the preponderant weight of the evidence. Rejected, as not in accordance with the preponderant weight of the evidence, and as not materially dispositive. 4.1. Rejected, as not materially dispositive. Rejected, as immaterial. Rejected, as immaterial. Accepted, but not a matter of factual dispute and immaterial. Respondent's Findings of Fact 1-10. Accepted. COPIES FURNISHED: Fred Seely, Executive Director Department of Professional Regulation Construction Industry Licensing Board P.O. Box 2 Jacksonville, FL 32202 Kenneth D. Easley, Esq. General Counsel Department of Professional Regulation 1940 N. Monroe Street, Ste. 60 Tallahassee, FL 32399-0792 E. Harper Field, Esq. Deputy General Counsel Department of Professional Regulation 1940 N. Monroe Street, Ste. 60 Tallahassee, FL 32399-0792 Mr. James Ilardi P.O. Box 8095 Jacksonville, FL 32239
