The Issue The issue for consideration in this case is whether Respondent’s certification as a teacher in Florida should be disciplined because of the matters alleged in the Administrative Complaint.
Findings Of Fact At all times pertinent to the issues herein, Petitioner was the state official responsible for the licensing and certification of teachers in Florida and for the regulation of the teaching profession in this state. Respondent was certified under Florida Educator’s Certificate Number 517092, covering the areas of Early Childhood Education and Elementary Education, with an English Speakers of Other Languages endorsement, through June 30, 1999. On or about April 4, 1995, Respondent was employed as a second-grade teacher at Tuttle Elementary School in Sarasota County, Florida. She was assigned homeroom number R5, which was contiguous to room R6, assigned to Ms. Frosch, also a second- grade teacher. Room R5 was the end room of four rooms in a row. Room D7 was assigned to Ms. Davenport, and Room D8 was assigned to Ms. Wajda. Respondent taught a self-contained second grade class made up of students, some of whom were learning disabled, some emotionally handicapped, and some who spoke English as a second language. These students took their special classes, (music, art, etc.), at a time other than when the continuous progress students took them. Tuttle Elementary School was scheduled to administer the National Achievement Test on April 4 - 18, 1995. Prior to April 4, Ms. Kurtin, an Assistant Principal at Tuttle, had scheduled a series of meetings in advance of the testing to brief the teachers and advise them of the ground rules for the testing process. Second grade teachers were scheduled to meet at 1:45 p.m. on Tuesday, March 22, 1995, and English Speakers of Other Language (ESOL) teachers were scheduled to meet at 3:00 p.m. on Wednesday, March 23, 1995. Respondent attended at least one of those meetings. At the meetings, the teachers were briefed on, among other matters, how to maintain security of the test booklets. Included was an injunction not to teach the students directly from the booklets in advance of testing. The test materials for the students were given to the individual teachers the day before the testing was to begin. The teachers had received their test manuals earlier and were instructed to safeguard them to insure the integrity of the test process. Testing for the second grade was to last between 25 to 40 minutes per day over a period of 6 to 7 days. Students in grades 1 through 3 were to place their answers directly into the test booklets. Students in grades 4 and 5 were provided answer sheets on which to place their answers. After each test session, the teachers were to collect the test booklets and, prior to turning them in for grading, go through them to erase or otherwise remove extraneous marks on the pages which might confuse the machine grading of the students’ test answers. After cleaning, the test booklets were to be secured until the next testing session or the end of the testing. One day during the testing period in the spring of 1995, Ms. Frosch was alone in her classroom while her students were at specials after the morning testing session. At approximately 10:15 a.m. she went outside her room for a break. When she went back inside, she heard Respondent talking with her class in the Respondent’s classroom. Though there is what appears to be a brick wall between Ms. Frosch’s room and that of the Respondent, Ms. Frosch contends she was able to hear what Respondent was saying, and it became clear to her that Respondent was reading exactly from the test booklet. In fact, Ms. Frosch took out her own booklet and was able to follow along with the Respondent. She heard Respondent read question 7, and heard the students give the answer. If a child gave the wrong answer, Respondent corrected the answer. Ms. Frosch, feeling that someone else should verify what was happening, went through the door which separated her room from that of Ms. Davenport, her team teacher. The desks of Ms. Frosch and Ms. Davenport were both near the doorway which joined their rooms. Frosch had Davenport come into her room and listen at the wall. Ms. Davenport heard Respondent reading word-for- word from the test booklet. By this time, Respondent was at question 12, and Ms. Davenport was able to follow along in Frosch’s test booklet. Though she only stayed to listen a short while, Davenport was shocked to hear what was going on because the material being discussed was on the test scheduled to be administered the next day. Ms. Frosch also got Ms. Wajda, from the next room down, and her aide, Ms. Salazar, to come to the room to listen also. Both Wajda and Salazar also heard Respondent reading the test questions aloud, word-for-word, to her students. By this time, Respondent was on page 11 of the booklet at questions 34 and 35. They heard Respondent read a question and the four possible answers, and then respond if the child selected the correct answer. Rather than use the in-room call button, Ms. Wajda went to get Ms. Kurtin, the Assistant Principal, at her office. She did this because in her opinion, it was inappropriate for Respondent to be doing what she was doing. Ms. Kurtin believed she was being taken to the Respondent’s room for a student discipline problem. She had no idea of the nature of the situation until she got there. When Ms. Kurtin reached Respondent’s room, she observed Respondent seated at her desk reading from what appeared to be a test booklet. Kurtin could not say that Respondent was reading from it verbatim. When questioned by Kurtin, Respondent said she was practice testing, but was changing the words in the questions. Ms. Kurtin advised respondent that it was inappropriate to use the real test booklets to practice from and to please put them away. Respondent did as Ms. Kurtin asked, but Kurtin felt nonetheless that she had to report the matter, which she did. Respondent has been a teacher since graduation from college in 1981. She started teaching at Tuttle at the second- grade level during the 1990-1991 school year. During the 1994- 1995 school year, the year in issue, she was a regular classroom teacher with 25 students of varying cultures and abilities. Only 6 of her students were regular students. The others were learning disabled, emotionally handicapped, or ESOL students. Respondent had worked with Ms. Frosch and some of the other teachers on teams for five years. She claims to be a soft- spoken individual and does not believe she could have been heard through the brick wall which separated her room from that of Ms. Frosch. She claims that as an African-American, she did not fully fit in with the other teachers, and her opinions and ideas were not given the same deference and consideration as those of the others. Notwithstanding that all of the other witnesses claimed to have experienced a congenial working, if not social, relationship with Respondent, she asserts that Frosh and Davenport would comment about her classroom discipline skills to the Principal who would, in turn, report the comments to her. Ms. Chester has administered the NAT many times before and knows the requirements for test security. On April 4, 1995, the day in issue, she arrived at school at the normal time. Because it was a test day and students normally pulled out were not taken, she had all her students with her in the room. For some reason not fully disclosed, her students were not to be tested until the next day. To spend the time productively, without following the regular lesson plan while the students were still coming in, she took her teacher’s manual, not, she claims, the actual test booklet, and started working with the children. She is adamant that she was not reading from the actual test manual. Though not made clear by the evidence at trial, it is most likely that she was reading from the teachers’ test booklet which, while not given to the students, contains the same materials and questions. Respondent is puzzled by several matters. She does not accept that the complaining teachers could have heard her as they claim, because she speaks so softly. There is, however, a door on either side of the room which joins it to the outside and to the adjoining room. She does not understand how she could have progressed from question 7 or 10, where she was first heard, to questions 34 or 35, where she was reading when Ms. Kurtin came in. It is not clear how much time passed from Frosch’s initial alert until the arrival of Ms. Kurtin. Finally, she does not know why the in-room call button was not used to summon Ms. Kurtin instead of Ms. Wajda going to get her in person. None of these questions has any material impact on the ultimate determination of the issue of whether Respondent was reading from the test manual. None of the teachers who testified on behalf of the Petitioner was of the opinion that the Respondent’s action in reading to her students from the test booklet was harmful to their mental health or physical safety. By the same token, no one opined that her actions exposed the students to unnecessary embarrassment or disparagement. No independent evidence was introduced by the Petitioner to support such allegations. It was accepted, however, and it is found, that Respondent’s use of the test booklet to practice with the children, regardless of her belief that the students would not remember enough to do them any good, was not professionally honest. It is irrelevant that no benefit to the Respondent could have resulted from the possibility that her students might have scored higher on the tests than they might have had she not read the questions to them.
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is recommended that the Commissioner of Education enter a Final Order finding Respondent guilty of exposing her students to conditions harmful to learning and of failing to maintain honesty in all professional dealings. It is further recommended that her certification as a teacher be placed on probation for one year, that she be reprimanded, and that she be required to take at least six hours of continuing education in the area of teacher ethics and testing procedures. DONE AND ENTERED this 13th day of April, 1998, in Tallahassee, Leon County, Florida. ARNOLD H. POLLOCK Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 SUNCOM 278-9675 Fax Filing (850) 921-6947 Filed with the Clerk of the Division of Administrative Hearings this 13th day of April, 1998. COPIES FURNISHED: Barbara J. Staros, Esquire 215 South Monroe Street Second Floor Tallahassee, Florida 32301 Stanley Marable, Esquire 677 North Washington Boulevard Sarasota, Florida 34236 Kathleen M. Richards Executive Director Educational Practices Commission 325 West Gaines Street Suite 224-E Tallahassee, Florida 32399-0400 Michael H. Olenick General Counsel Department of Education The Capitol, Plaza Level 08 Tallahassee, Florida 32399-0400 Iris Anderson, Program Specialist Procedural Safeguard Department of Education Bureau of Education for Exceptional Students 325 West Gaines Street, Suite 614 Tallahassee, Florida 32399-0400 Jerry W. Whitmore, Program Director Professional Practices Services Department of Education 325 West Gaines Street Suite 224-E Tallahassee, Florida 32399-0400
Findings Of Fact The Petitioner, James Ilardi, was an unsuccessful candidate for the June, 1989 General Contractor's Construction Examination in the State of Florida. He is an experienced contractor and is licensed in the State of South Carolina. He is the past President of the Charleston Contractor's Association in South Carolina. He has experience with most types of building construction, including office buildings, military facilities, hospitals, factories and other large projects. He served as Chief Executive Officer for a design and construction firm for a period of ten years. The Respondent is an agency of the State of Florida charged with administering the certified general contractor's examination and with regulating the licensure and practice of construction contractors in the State of Florida. The Petitioner sat for the certified general contractor's examination in June, 1989. He has challenged the scoring of his answers to questions 2, 11, 12, 17, 19 and 33 on that examination. During the course of the hearing, he abandoned his challenges to questions 2, 17, 19 and 33. If he were accorded correct answers to either of the remaining challenged questions, numbers 11 or 12, he would have a sufficient score to obtain a passing grade of 70 on that examination. Both questions 11 and 12 used a "critical path network diagram" for use in working out the correct answer to the questions. The Petitioner criticized the diagram as being obscure, difficult to read and containing error. He maintained that it was not supported by the representations found in the reference materials recommended by the Respondent, in its "Instructions to Candidates", as being the material to use to arrive at answers to the questions. The Petitioner contends that the size of the diagram, "the multiple fonts, the difference in the intensity of the print, and the use of symbols all contribute to the obscurity and illegibility of the diagram, itself". In particular, he complains that the symbol listing includes a symbol which he did not find on the diagram. That is, the symbol for "structural steel" and "steel bar joists, which is two straight vertical parallel lines. He also complains that general practice in the construction industry, in his experience, and as indicated in the reference work "Construction Contracting", pages 325-326, one of the references listed for candidates to use in answering these questions, recommends against the use of symbols in lieu of abbreviated notations for description of activities on such a diagram. The main complaint he had concerning the use of symbols, however, was the fact that use of symbols, and having to constantly defer to the symbol legend on the exam materials, was time-consuming and was not generally accepted industry practice or procedure. He contends that the diagram contains error or is obscure and does not conform to the Respondent's recommended reference materials nor to industry standards and. is deficient in format, design and reproductive quality. Thus, he maintains that questions 11 and 12 do not adequately test the knowledge or skills necessary for licensure as a general contractor. The Petitioner acknowledged that the questions at issue had been reviewed twice by the Respondent's examination content specialist and that an "item analysis and review process" by the Respondent's expert resulted in the Respondent maintaining its position that the two questions and supporting materials were valid in fairly testing the knowledge of general contractor licensure candidates. In summary, the Petitioner contends that as to question 11, the symbol for steel bar joists, the two parallel vertical lines, does not appear on the diagram; therefore, he was unable to determine whether his answer was correct or not. As to question number 12, he maintains, in essence, that the use of symbols instead of brief abbreviated descriptions of the activities involved, accompanying the arrows in the diagram which indicate the critical path for the activity in question (paint work), render answering the question confusing and time consuming in having to constantly refer to the symbol legend and look for the symbols. He states that, in his 20 years of construction industry experience, he has not had to use symbols in working with a critical path diagram. The Petitioner did not demonstrate, however, that the use of symbols was incorrect procedure as delineated in the reference materials supplied to the candidates and which they were instructed to use in answering the questions on the examination. The Respondent produced the testimony of Mr. Olson, a Florida certified general contractor, who is also employed with the National Assessment Institute which developed this examination. Mr. Olson, however, did not, himself, have a hand in developing the examination. Mr. Olson did, however, review the Petitioner's challenges to the questions at issue and his responses, reviewed questions 11 and 12, as well as the Respondent's asserted correct answers to those questions and the methodology used in reaching those answers. Mr. Olson established that this was an "open-book" examination and the candidates were informed of and supplied all necessary reference materials to answer these two questions. The only optional consideration was that candidates could have used a calculator to speed up their calculations and were informed that it was permissible to use a calculator. Question 11 required candidates to calculate the total time necessary to install structural steel and steel bar joists in interpreting the activity network represented by the diagram in question. They were asked to calculate whether the installation was ahead of schedule or behind schedule and by how much. Mr. Olson established that the correct response was "C", which is two days behind schedule. Mr. Olson demonstrated that it was quite possible for a candidate to make this calculation and track this in formation on the diagram provided the candidates, through reading the path with the symbols, which alphabetically represent the activity, and which are numerical in representing the time in days. He established that this is very typical of the construction industry, related to the preparation, reading and interpreting of blueprints. A tremendous amount of symbols and legends are typically used in preparing and interpreting blueprints. Mr. Olson established that the pertinent number, 85 days, could be calculated for installation of structural steel and steel bar joists, based upon the information supplied to the candidates. By using the diagram and the information supplied with the question, the candidate can calculate that the actual number of days that were taken for the job was 87 days and therefore, that the project, at that point, was two days behind schedule. Mr. Olson performed this calculation by using the actual diagram the Petitioner used and reference information the Petitioner was given to use in answering the actual examination question at issue. He also established that the two parallel lines representing steel bar joists and structural steel on the diagram, and in the symbol legend supplied with the diagram, were indicated on the diagram supplied to Mr. Ilardi at the examination. Mr. Olson also established that the reference quoted for question number 11 was walkers Building Estimator's Reference Book, which, indeed, listed the type of activity network depicted in the diagram used by candidates for question number 11 and 12. Mr. Olson also established that question number 12 requires a candidate to work through an activity network diagram to find the amount of days necessary from the beginning of a project to the time the painting activity begins. He established that the answer could be obtained without the use of any other reference materials other than the information depicted on the diagram, itself, associated with the question. He established that the only correct answer from that information on the diagram could be "D" or 153 days. The Petitioner did not establish that his answer to question number 12, nor to question number 11 for that matter, was a correct answer and did not establish that there was any misleading quality or ambiguity in the wording of the questions and the associated information which would mislead a candidate into calculating the wrong answers or that there was erroneous information depicted in the reference materials or the diagram which would result in the candidate being misled into giving a wrong answer to questions 11 and 12.. Mr. Ilardi challenged the examination as to the testing environment, as that relates to the ambient light level in the examination room and to the acoustic qualities of the room. He also asserted that the test was not standardized throughout the State and was biased due to age, because of the perceived hearing and vision difficulties which he believed were caused by the acoustics in the examination room and the light available. Other than stating his opinions in this regard, he produced no testimony or evidence concerning these alleged qualities of the testing environment. It was demonstrated by she Respondent that, indeed, the test is standardized throughout the State and is the one given to all candidates in Florida, regardless of the test location.
Recommendation Having considered the foregoing Findings of Fact, Conclusions of Law, the evidence of record, and the candor and demeanor of the witnesses, it is therefore RECOMMENDED that a Final Order be entered denying Petitioner's request to receive a passing grade on the certified general contractor's licensure examination. DONE and ENTERED this 4th day of May, 1990, in Tallahassee, Leon County, Florida. P. MICHAEL RUFF Hearing Officer Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-1550 (904)488-9675 Filed with the Clerk of the Division of Administrative Hearings this 4th day of May, 1990. APPENDIX TO RECOMMENDED ORDER CASE NO. 89-3784 Petitioner's Proposed Findings of Fact Rejected. The Petitioner was not qualified as an expert witness. Other than that, this finding is accepted. Rejected, as not in accord with the preponderant weight of the evidence. Rejected, as not supported by the preponderant weight of the evidence. 4.A.-4.C. Accepted. 4.D. Rejected, as not in accordance with the preponderant weight of the evidence. Accepted, but not itself materially dispositive. Rejected, as not in accordance with the preponderant weight of the evidence. Rejected, as not in accordance with the preponderant weight of the evidence. Rejected, as not in accordance with the preponderant weight of the evidence and as subordinate to the Hearing Officer's Findings of Fact on this subject matter. Rejected, as not constituting a finding of fact, but rather a quotation from the transcript of the proceedings. Rejected, as not in accordance with the preponderant weight of the evidence and as subordinate to the Hearing Officer's Findings of Fact on this subject matter. Rejected, as immaterial and not probative of the issues of whether the questions were ambiguous or misleading or whether the Petitioner's answers were correct. Rejected, as not in accordance with the preponderant weight of the evidence. Rejected, as not in accordance with the preponderant weight of the evidence. Rejected, as not in accordance with the preponderant weight of the evidence, and as not materially dispositive. 4.1. Rejected, as not materially dispositive. Rejected, as immaterial. Rejected, as immaterial. Accepted, but not a matter of factual dispute and immaterial. Respondent's Findings of Fact 1-10. Accepted. COPIES FURNISHED: Fred Seely, Executive Director Department of Professional Regulation Construction Industry Licensing Board P.O. Box 2 Jacksonville, FL 32202 Kenneth D. Easley, Esq. General Counsel Department of Professional Regulation 1940 N. Monroe Street, Ste. 60 Tallahassee, FL 32399-0792 E. Harper Field, Esq. Deputy General Counsel Department of Professional Regulation 1940 N. Monroe Street, Ste. 60 Tallahassee, FL 32399-0792 Mr. James Ilardi P.O. Box 8095 Jacksonville, FL 32239
The Issue The issue in this case is whether Florida Administrative Code Rule 61G15-21.004(2) is an invalid exercise of delegated legislative authority.
Findings Of Fact The Parties Petitioner is an applicant for licensure as a professional engineer ("P.E.")2 in Florida, and currently works in the discipline of environmental engineering in Florida. His practice focuses primarily on water-related areas within that discipline. Petitioner is not currently licensed as a P.E. Respondent is a board within the Department of Business and Professional Regulation ("Department"). It is the state agency created pursuant to section 471.007, Florida Statutes, and charged with licensing professional engineers in Florida. Respondent is vested with the authority to adopt rules to implement chapter 471, regarding the regulation of the practice of engineering in Florida, as defined in section 471.005(7). Respondent adopted the Challenged Rule at issue in this proceeding. Statutory and Rule Background The engineering profession in Florida is regulated pursuant to chapter 471. A person may become licensed as a P.E. in Florida by applying for licensure, fulfilling specified educational and experience requirements, and 2 For purposes of this Final Order, the terms "professional engineer" or "P.E." will be used to refer to persons who are licensed engineers under chapter 471, Florida Statutes. either being endorsed for licensure as provided in sections 471.015(3) and (5), or passing the required licensure examinations. § 471.015, Fla. Stat. Pursuant to section 471.015, Respondent has adopted Florida Administrative Code Chapter 61G15-20, which codifies, in rule, the requirements for licensure as a P.E. in Florida. An applicant for licensure must be a graduate of a Board-approved engineering program; have the requisite number of years of engineering experience; and have passed the specified licensure exams. Fla. Admin. Code R. 61G15-20.0010. Section 455.217(1)(d) authorizes Respondent to adopt, by rule, the use of a national professional licensing examination that the Department has certified as meeting requirements of national examinations and generally accepted testing standards. To implement section 455.217(1), Respondent has adopted rule 61G15- 21.001, titled "Examination Designated; General Requirements." This rule requires that, unless an applicant qualifies for licensure by endorsement, he or she must pass the National Council of Examiners for Engineering and Surveyors ("NCEES") licensure exam. Part I of the NCEES exam is the Fundamentals of Engineering ("FE") exam, and part II of the NCEES exam is the Principles and Practice Exam ("PP") exam. Respondent has entered into a contract with NCEES to provide the FE and PP exams in Florida. A person must pass both the FE and PP exams to be licensed as a P.E. in Florida. § 471.015(1), Fla. Stat. The Challenged Rule states: "[t]he passing grade for Principles and Practice Exam is determined by the National Council of Examiners for Engineering and Surveying, where psychometric statistical methods are used to determine the level of performance that corresponds with minimal competence in the discipline." Fla. Admin. Code R. 61G15-21.004(2). The Challenged Rule is a subsection of rule 61G15-21.004, which is titled "Passing Grade." The Challenged Rule specifically and exclusively addresses the method for determining the passing grade on the PP exam. Sections 455.217(1)(d) and 471.013 are cited as the rulemaking authority for the Challenged Rule, and sections 455.217(1)(d) and 471.015(1) are cited as the law implemented by the Challenged Rule. The term "engineering," as used in section 471.005(7), includes the term "professional engineering," and defines the types of services and creative work that constitutes "engineering." An "engineer," as defined in section 471.005(5), includes the terms "professional engineer" and "licensed engineer," and means a person who is licensed to engage in the practice of engineering under chapter 471. By contrast, an "engineer intern," as defined in section 471.005(6), means a person who has graduated from a Board-approved engineering curriculum and has passed the FE exam. By definition, these are distinct terms. The term "engineer" is used to describe a person licensed as a P.E. under chapter 471, while the term "engineer intern" is used to described a person who may engage in the kinds of activities described within the term "engineering," as defined in section 471.005(7), but who is not licensed as a P.E. in Florida, and, therefore, is not authorized to hold himself or herself out as a licensed engineer in Florida. The Rule Challenge Petition The Rule Challenge Petition alleges four grounds under section 120.52(8) for invalidating the Challenged Rule. Alleged Invalidity of Challenged Rule under Section 120.52(8)(c) In paragraph 15 of the Rule Challenge Petition, Petitioner asserts that section 455.217(1)(c)—which, at the time the Rule Challenge Petition was filed, was cited as the specific authority for, and law implemented by, the Challenged Rule—did not authorize the Challenged Rule, so that the Challenged Rule enlarged, modified, or contravened the specific provisions of law implemented, pursuant to section 120.52(8)(c). As a result of Petitioner having filed the Rule Challenge Petition, Respondent discovered that it had not updated its citation of the specific authority for, and law implemented by, the Challenged Rule, when section 455.217(1) was amended and renumbered in 1997, so that section 455.217(1)(c) no longer was the correct citation to the law implemented by the Challenged Rule. Respondent requested the Department of State, Administrative Code and Register Section ("DOS"), to make a technical, non-substantive change to the Challenged Rule. As authorized by Florida Administrative Code Rule 1-1.010(10), DOS updated the statutory citation to section 471.217(1)(d), which is the correct citation to the law implemented by the Challenged Rule. This technical change nullifies the alleged invalidity ground set forth in paragraph 15 of the Rule Challenge Petition, and Petitioner concedes this. In paragraph 16 of the Rule Challenge Petition, Petitioner also alleges that the Challenged Rule enlarges, modifies, or contravenes section 455.217(1)(a), because the PP examination does not adequately and reliably measure an applicant's ability to practice the profession regulated by the Department. However, as discussed below, section 455.217(1)(a) is not cited as a specific provision of law implemented by the Challenged Rule, so cannot form the basis of a challenge to the Rule under section 120.52(8)(c). Alleged Invalidity of Challenged Rule under Section 120.52(8)(d) In paragraph 17 of the Rule Challenge Petition, Petitioner alleges that the Challenged Rule is invalid under section 120.52(8)(d) because it is vague, fails to establish adequate standards for agency decisions, or vests unbridled discretion in the agency. In support of this alleged invalidity ground, Petitioner asserts that the Challenged Rule is vague because "the level of performance on the PP exam is stated to correspond with minimal competency, yet there are no rules which provide definitive guidance to NCEES on what constitutes the general areas of competency in regards to engineering practice."3 Petitioner also alleges, in paragraph 19 of the Rule Challenge Petition, that the Challenged Rule is invalid under section 120.52(8)(d) because it equates passage of the PP exam with a 30-year practice experience requirement for licensure by endorsement set forth in section 471.015(5)(b). To this point, Petitioner states: "I can't think of anything more arbitrary than the principles and practice exam equating to near[-]retirement level experience."4 Alleged Invalidity of Challenged Rule under Section 120.52(8)(e) In paragraph 18 of the Rule Challenge Petition, Petitioner alleges that the Challenged Rule is invalid pursuant to section 120.52(8)(e) because it is arbitrary, for several reasons. Paraphrased, these reasons are: passage of the PP exam does not accurately reflect, or equate to, minimal competence in the discipline; the PP Exam does not accurately evaluate an individual's engineering ability level, but instead evaluates an individual's exam performance compared to average group exam performance; the PP exam does not reliably distinguish between minimal competence and incompetence to practice engineering, as evidenced by the fact that engineers who fail the PP exam still competently perform, and, thus, keep, their engineering jobs; passing the PP exam, by itself, does not certify an individual to competently perform any engineering service or creative work as defined in 3 Section 455.217(1)(b) requires, for each exam developed by the Department or a contracted vendor, that the general areas of competency covered by the exam be specified by rule. The last sentence of that subsection states that the requirements of subsection (b) do not apply to national exams, such as the NCEES PP exam, which are approved and administered pursuant to section 455.217(1)(d). Thus, the law implemented by the Challenged Rule does not require areas of competency to be specified in the Challenged Rule. 4 Because paragraph 19 of the Rule Challenge Petition alleges that the rule is arbitrary, the undersigned considers this paragraph to constitute a challenge to the Challenged Rule under section 120.52(8)(e), rather than under section 120.52(8)(d), as cited in the Rule Challenge Petition, and has addressed this ground in the Conclusions of Law section dealing with that alleged invalidity ground. section 471.005(7); the PP exam does not reliably determine if an examinee is minimally competent, due to an incorrect reference point; and the PP exam does not reliably distinguish between individuals whose practice of engineering would protect the public health and safety and those whose practice of engineering would constitute a danger to public health and safety. Alleged Invalidity of Challenged Rule under Section 120.52(8)(f) In paragraph 20 of the Rule Challenge Petition, Petitioner alleges that the Challenged Rule is invalid pursuant to section 120.52(8)(f) because it imposes regulatory costs on the regulated person, county, or city, which could be reduced by the adoption of less costly alternatives that substantially accomplish the statutory objectives. To this point, Petitioner proposes a state- conducted investigation of an applicant as a substitute for the PP exam, and contends that "there's a premium associated with a national exam. It's also clear that the regulatory cost imposed on me and all future examinees could be substantially reduced if the Board conducted their exam as an investigation and did not incorporate the NCEES exam." However, as more fully discussed below, this challenge ground is time- barred by section 120.541(1)(g), and, therefore, is not a legally-cognizable basis for invalidating the Challenged Rule in this proceeding. The Parties' Stipulated Facts Petitioner is an Engineer Extern, Texas EIT 56990. Pursuant to section 471.015(1), the Florida Engineers Management Corporation "shall issue a license to any applicant who the Board certifies is qualified to practice engineering and who has passed the [FE] exam and the [PP] exam." Both the FE exam and the PP exam are created by NCEES, pursuant to section 455.217(1)(d), which states, in pertinent part: "a board . . . may approve by rule the use of any national examination which the [Department of Business and Professional Regulation] has certified as meeting the requirements of national examinations and generally accepted testing standards pursuant to department rules." Pursuant to section 471.015(1), Petitioner passed the FE exam on September 3, 2016. Passing the FE is a prerequisite to take the PP exam. Petitioner registered to take the NCEES computer-based PP exam for environmental engineering on April 22, 2020. He paid the $350 exam fee and additional monies for test preparation material, and spent at least 100 hours preparing for the exam. He was notified, on April 6, 2020, by Pearson Vue, the test center company, that the exam was cancelled due to the Covid-19 pandemic. On April 7, 2020, Petitioner registered to take the exam on July 15, 2020, which was the earliest available date for taking the exam in his local area. On April 9, 2020, Petitioner canceled his registration for the July 15, 2020, exam and decided to apply for licensure as a P.E. without passing the PP exam. Pursuant to section 471.015(2)(a)1., on May 18, 2020, Petitioner filed his application for licensure with the Board. The application provides information stating that Petitioner meets the requirements of section 471.013(1)(a)1., and has at least four years of active engineering experience of a character indicating competence to be in responsible charge. In compliance with the education requirements of section 471.013(1)(a)1., Petitioner earned a Bachelor of Science degree in Chemical Engineering from the University of Florida, which is accredited by ABET. Petitioner provided an official transcript to the Board for verification purposes. In compliance with rule 61G15-20.002, Petitioner states in his application that he has at least four years of acceptable engineering practice, consisting of one year of experience equivalent through his Master of Engineering degree from the University of Texas at Austin, an ABET- accredited institution; and over three years of professional engineering work verified by licensed engineers. According to rule 61G15-20.002, an applicant must list three current personal references who are professional engineers. Thirteen licensed engineers submitted documentation to the Board regarding Petitioner's type of qualifying experience, level of engineering competency, and professional integrity. All references circled "yes" to the question "would you employ the applicant in a position of trust?" Two additional references in the engineering industry also provided their opinion on Petitioner's integrity and competence. On June 1, 2020, Petitioner received notification from the Board that his Florida 0901 1031-P.E. Endorsement application was incomplete. Petitioner addressed all items listed in a timely fashion to participate in the June 10, 2020, Board meeting. The Board stated during an informal hearing on June 10, 2020, that Petitioner would not be granted P.E. licensure due to not having passed the PP Exam. Petitioner is substantially affected by the Challenged Rule because it disqualifies him from becoming a licensed engineer in Florida without passing the PP exam. Findings of Fact Based on Evidence Presented at the Final Hearing A. Findings Regarding the Evidence Presented in Petitioner's Case Testimony of Witnesses Hoot and Grace David Hoot and Nigel Grace, both of whom are licensed professional engineers, testified regarding Petitioner's abilities and skills as a practicing engineer. Neither Hoot nor Grace was qualified as an expert in the field of psychometrics or related topics. Therefore, any opinions regarding these subject matters to which they testified at the hearing have been treated as personal opinions, rather than expert opinions. Hoot characterized Petitioner as a good, diligent young engineer, and described Petitioner's role in various projects on which they worked together. Based on his work with Petitioner, it was Hoot's personal opinion that Petitioner possesses the integrity and competence to work as a licensed engineer who would serve the public health and safety. Hoot testified that when he took the PP exam approximately 38 years ago, it was a free response exam. He stated that he was "not exactly sure" that a multiple-choice exam captures an examinee's ability to apply reason and judgment, but he acknowledged that he does not know how the PP exam currently is developed. He offered his personal opinion that it was possible for a competent engineer to fail the PP exam. Hoot also offered his personal view that engineering licensure constitutes a standard of care; however, he did not think anything provides a guarantee of competence. It was his personal view that experience tended to make one more competent. He also offered his personal view that as an engineer gains more experience and becomes more specialized, it is understandable that he or she would not score as well as an engineer who is gearing up toward the four-year experience goal which enables them to apply for licensure as a P.E. As Hoot put it, "life happens. You have children involved. You move jobs. I think you . . . maybe have less time to study. . . you get farther away from the study habits of . . . learning to be able to take tests." Grace, who is employed as a P.E. with Brown and Caldwell ("B&C"), a large U.S. engineering firm, testified regarding Petitioner's work experience while he was employed by B&C. Petitioner's experience included working on drinking water projects, utilities, upgrading process equipment, site evaluations, bench scale testing, and other projects. Based on Grace's work with Petitioner, it was his personal opinion that Petitioner possesses the integrity and competence to work as a licensed engineer who would serve the public health and safety. Grace took the PP exam approximately 28 years ago, and at the time, a major portion of the exam consisted of long-form written exam questions that provided the opportunity for examinees to provide free response answers and earn partial credit for partially-correct answers. He testified that the exam also had a multiple-choice component. Grace testified that, "based on instinct," he knows well-designed multiple-choice questions can provide the same insight into an examinee's decision-making judgment as long-form questions. Grace's personal view is that passing the PP exam does not guarantee competence, and it is possible for an engineer to be competent in some engineering disciplines but not others. Grace also agreed that it was logical that an examinee with approximately four years' experience would perform better on the PP exam because he or she would have better-honed test-taking skills and be fresher in some areas tested on the exam. By contrast, engineers who have practiced longer have more experience, but often have become specialized and further removed from the test-taking environment. Thus, passing a broad-spectrum exam becomes a bigger hurdle for engineers who have practiced longer. Petitioner's Testimony Petitioner testified on his own behalf at the final hearing. As discussed above, Petitioner holds a bachelor of science degree in chemical engineering from an ABET-accredited institution, and holds a master of engineering degree from an ABET-accredited institution. Although Petitioner, through his training as an engineer, is skilled at mathematics and statistics, he is not trained, and does not have any substantial experience, in the field of psychometrics.5 5 As discussed in greater detail below, psychometrics is a specialized field of study that concerns the theory and technique of objective psychological measurement of skills, knowledge, abilities, and educational achievement. Petitioner acknowledged that he does not have a degree in psychometrics and that he is not trained in, or knowledgeable about, preparing and administering high-stakes professional examinations. At the time he filed the Rule Challenge Petition, Petitioner had no personal knowledge of the work done by psychometricians; did not know what a "cut score" was; and was not familiar with the Modified Angoff Method, item response theory, the specifics of converting raw scores to scaled scores, or other any psychometric tools and methods employed to prepare and score the PP exam. Petitioner acquired some rudimentary knowledge in a few of these areas in preparing for the final hearing in this proceeding.6 Section 90.701(2), Florida Statutes, prohibits a lay witness from testifying as to an opinion regarding a matter involving specialized knowledge.7 Accordingly, it is determined that Petitioner is not competent to provide an expert opinion regarding psychometrics and related areas, such as item response theory. He was not qualified, tendered, or accepted at the final hearing as an expert in psychometrics or any related areas. Because Petitioner was not qualified to testify as an expert witness at the final hearing, Petitioner's testimony regarding psychometrics, item response theory, scaled scores, the Modified Angoff Method, high-stakes professional testing, and all other specialized subject matters, consisted of opinion testimony by a lay witness. The only instances in which a lay witness 6 Petitioner does not have any special knowledge, formal training, education, or experience in the specialized field of psychometrics. His knowledge about these areas was acquired by reading and study in preparation for the final hearing. Because Petitioner lacks special knowledge, experience, training, and education in psychometrics, he is not qualified to testify as an expert in psychometrics or related topics, such as item response theory. See Chavez v. State, 12 So. 3d 199, 205 (Fla 2009)(in determining whether a witness is qualified to render an opinion as an expert in a specialized field, the court must determine whether the witness is adequately qualified to render an opinion based on special knowledge, experience, training, or education). 7 Chesser v. State, 30 So. 3d 625 (Fla. 1st DCA 2010)(it is error for a court to accept opinion testimony of a lay witness in a specialized subject matter area). may provide opinion testimony are when the lay witness's opinion is based on firsthand knowledge through personal perception.8 As the undersigned explained during the final hearing, rather than excluding Petitioner's lay opinion testimony, she would consider these pertinent evidentiary principles in determining the weight to be afforded Petitioner's lay opinion testimony in this proceeding. Applying these evidentiary principles, it is determined that Petitioner's testimony regarding psychometrics and related topics, such as item response theory, classical response theory, and high-stakes test reliability and validity, concerned specialized subject matters not within the realm of common knowledge or based on Petitioner's personal perception. Rather, such specialized subject matters required expert witness testimony, pursuant to section 90.702, and as addressed above, Petitioner was not shown to be an expert in any of these specialized subject matters. Because Petitioner's testimony constituted the type of opinion testimony that is not permissible by a lay witness, pursuant to section 90.701, such testimony is not afforded weight in this proceeding. Petitioner testified that "minimal competence," which is the standard measured on the PP Exam, equates to "competence" as defined in the dictionary—that is, the "sufficient ability for a specific need or requirement." However, this position ignores that, for purposes of the Challenged Rule, "minimal competence" is a term of art specifically used, in the psychometric measurement context, to describe the level of competence that corresponds to a passing score on the PP exam. As more fully discussed below, the PP exam is developed and scored using psychometric tools and methods. 8 Nat'l Commc'ns. Indus., Inc. v. Tarlini, 367 So. 2d 670, 671 (Fla. 1st DCA 1979)(lay witness testimony regarding a specialized subject matter was not admissible into evidence because the testimony was not regarding a subject matter about which the witness could testify based on common knowledge or his personal perception.) Petitioner contends that engineering experience is, by itself, a reliable measure of competence, so it is unnecessary to also pass the PP exam. Petitioner testified "the application process and [PP] exam have two entirely different methods to identify the same result: whether an engineer in training is competent enough to become a licensed engineer It is not logical for two checks of competence to come up with different results. There should be consistency." In support of this position, Petitioner relies on section 471.015(2)(a), which requires "at least 4 years of active engineering experience of a character indicating competence to be in 'responsible charge' of engineering." § 471.015(2)(a), Fla. Stat. "Responsible charge" is defined in rule 61G15-18.011(1) as the degree of control an engineer is required to maintain over engineering decisions made personally or by others over which the engineer exercises supervisory direction and control authority. An engineer in responsible charge is the "engineer of record," as defined in rule 61G15-30.002(1). Rule 61G15-30.002(1) defines "engineer of record" as a Florida professional engineer who is in responsible charge. Thus, an engineer who is qualified, for purposes of being in responsible charge pursuant to section 471.015(2)(a) must, in addition to having the minimum statutory experience, be a licensed P.E. This means that he or she necessarily must have passed the PP examination. These statutory and rule provisions collectively reinforce the point that for an engineer to demonstrate competence for purposes of holding himself or herself out as an "engineer," as defined in section 471.005(7), he or she must satisfy all three requirements of section 471.015(2)(a)—i.e., education, experience, and passing the licensing exam.9 9 The requirement to meet these three requirements, including the PP exam, is codified in section 471.051(2)(a). Eliminating the exam requirement and relying strictly on education and/or experience for licensure would require the Florida Legislature to amend this statute Petitioner echoed the testimony of Hoot and Grace that licensure is not a guarantee of competence, and that passing the PP exam does not guarantee minimal competence. To this point, he testified that he does not believe that the PP exam adequately and reliably measures an applicant's ability to practice engineering, and that experience is a better indicator of competence than passing the exam. By way of example, Petitioner described his own experience10—which he characterized as "directly matching" the activities in which a licensed engineer engages—and compared that experience to measuring competence by an exam, which Petitioner characterized as "attempt[ing] to indirectly measure my ability as an engineer." Based on his personal experience, Petitioner contends that experience better demonstrates competence to be licensed as a P.E.; that passing the PP exam does not indicate minimal competence to practice engineering; and that failing the PP exam does not mean that the examinee is not minimally competent. He further testified that examinees who fail the PP exam likely are minimally competent, since the engineering jobs they hold when applying for licensure likely would require that they be minimally competent in order to have been hired.11 to eliminate the exam requirement. The undersigned is not authorized by statute or the Florida Constitution to eliminate the PP exam requirement for licensure under chapter 471. 10 Petitioner's experience, set forth in his P.E. licensure application, was verified by his supervising engineers. 11 Petitioner appears to conflate being determined not "minimally competent" for purposes of passing the PP exam, with "incompetence," which is defined in Florida Administrative Code Rule 61G15-19.001(5) as the "physical or mental incapacity or inability of a professional engineer to perform the duties normally required of the professional engineer." Part of this confusion may be due, in part, to Respondent's response to one of Petitioner's interrogatories asking for a definition of "minimal competence." Rather than directly answering the interrogatory, Respondent referred Petitioner to the definition of "incompetence" for purposes of imposing discipline under Respondent's disciplinary rules—thus causing Petitioner to understandably assume that failing to demonstrate minimal competence through passing the PP exam equates to "incompetence," as defined in rule 61G15-19.001(5). However, the fact that Petitioner has not demonstrated "minimal competence" on the PP exam does not mean that he is incompetent; it simply means that he has not yet passed the PP exam for licensure as a P.E. in Florida. To this point, if failing to demonstrate "minimal competence" by passing the PP exam equated to being incompetence, every person who performs engineering work in To further illustrate this point, Petitioner noted that the data regarding passage rate of the PP exam shows that examinees having zero years of experience are almost twice as likely to pass the PP exam as examinees having 11 or more years of experience. However, as Hood and Grace explained, and as further discussed below, engineers having more than four to five years of experience begin to specialize in narrower fields and "life happens," in that personal and professional circumstances render it more difficult to prepare for and take a high-stakes test. Petitioner also disputed the accuracy of the PP exam preparation and scoring process. In particular, he took issue with the "model law engineer" standard to which the exam is designed. As discussed more extensively below, this standard equates to the competence level of an engineer having four years of engineering experience and who is capable of practicing engineering in a manner that protects the public health and safety. In particular, Petitioner contends that designing the PP exam to the "model law engineer" standard is unfair to anyone taking the exam that does not have exactly four of years of engineering experience. Notably, however, section 471.015(1), which is the statute implemented by the Challenged Rule, establishes four years as the engineering experience required for licensure as a P.E. Thus, the "model law engineer" standard is rationally related to the statutory minimum experience level for purposes of demonstrating minimum competency to be licensed. Petitioner also contends that the subject matters tested on the PP exam are unfairly broad, so that engineers who specialize in a particular area within an engineering discipline—such as specializing in water-related areas in environmental engineering—are disadvantaged by being required to take Florida but has not passed the PP exam would be "incompetent," and, thus, potentially subject to disciplinary action. an exam that covers a broad range of areas beyond his or her area of specialty. Petitioner further contends that it is irrational to test an examinee on particular areas that are irrelevant to his or her work and/or desired career path. However, the PP exam for a particular discipline is specifically designed to ensure that a licensed P.E. is competent to practice over a range of specific areas encompassed within that particular discipline. This is because once a person becomes a licensed P.E., he or she may practice engineering within any discipline or specific area within that particular discipline, subject to the professional and ethical requirements to limit practice to the disciplines and areas in which the engineer is actually competent. Thus, the breadth of the PP exam is designed to help ensure minimal competence to practice engineering in a manner that protects the public health and safety. Petitioner also contends that because the PP exam for some engineering disciplines tests a broader range of areas than the PP exam may test for other disciplines, the exam inconsistently measures minimal competency across the range of engineering disciplines. However, as discussed in detail below, the subject matters tested on the PP exam for a given discipline are chosen by subject-matter experts who are licensed engineers practicing in that particular engineering discipline, and are deemed, by those subject matter experts, to be most important to test for purposes of measuring competency in that discipline. Thus, while the number of discrete subject matters tested on the PP exams may differ across the various engineering disciplines, this difference is, factually and logically, a function of expert consensus regarding which subject matters need to tested to demonstrate minimal competence. Petitioner also contends that the Challenged Rule is vague because it does not specifically identify the disciplines, and the areas within each discipline, that are tested on the PP exam. Respondent has contracted with NCEES to be responsible for preparing, administering, and scoring the PP exams, pursuant to section 455.217(1)(d). NCEES conducts a methodical process, discussed in detail below, to determine the specific disciplines for which to develop a PP exam and the areas to be covered on the PP exam for a discipline. Exam specifications are then developed by subject matter experts within that discipline, and are published by NCEES. These specifications inform prospective examinees regarding the particular areas that will be tested on the PP exam for the discipline, and the number of questions for each specific area that will appear on the exam. Thus, prospective examinees are not left to wonder or guess about which disciplines will be tested; the areas within each discipline that will be tested; or the relative weight that will be assigned to each area tested.12 Although the Challenged Rule does not identify the specific disciplines tested on the PP exam, rule 61G15-21.001(1)(b)—which actually adopts the PP exam as an engineering licensure exam in Florida—states that the PP exam "is given by discipline." Therefore, even if section 455.217(1)(d) required the specific areas of competency to be identified by rule, such areas would have been identified in rule 61G15-21.001(1), rather than in the Challenged Rule.13 12 See Cole Vision v. Dep't of Bus. And Prof'l Reg., 688 So. 2d 404, 410 (Fla. 1st DCA 1997)(a rule is impermissibly vague if it is drafted in terms so vague that men of common intelligence must necessarily guess at its meaning or application. 13 Neither sections 455.217(1)(d) nor 471.015(1) specifically authorize or require Respondent to adopt rules identifying the general areas of competency tested on the PP exam. By contrast, exams developed by the agency pursuant to section 455.217(1)(b) must identify, by rule, the general areas of competency to be tested. Had the Legislature intended for exams authorized under section 455.217(1)(d)—of which the PP exam is an example—to adhere to the same requirement, the statute would have so stated. See Pro-Art Dental Lab, Inc. v. V- Strategic Grp., LLC, 986 So. 2d 1244, 1258 (Fla. 2008)(the specific mention of one thing in a statute implies the exclusion of another). Furthermore, section 120.54(1)(g) expressly requires a rule to address only one subject. Thus, if the Challenged Rule also addressed the areas of competency to be tested on the PP exam, it would violate section 120.54(1)(g). Petitioner also asserted, at the final hearing, that Respondent did not certify the PP exam as meeting the requirements of national examinations and generally accepted testing standards pursuant to department rules, as required by section 455.217(1)(d). However, Petitioner did not raise this alleged invalidity basis in the Rule Challenge Petition, so he is foreclosed from raising and litigating it at the final hearing. See § 120.56(1)(b), Fla. Stat. Petitioner also testified that, in transitioning from paper-and-pencil PP exams to computer-based exams, NCEES is relying on two different theories—classical test theory and item response theory—and that this reliance does not comport with generally accepted testing standards. However, as discussed above, Petitioner was not qualified as an expert in the specialized area of high-stakes examination preparation and scoring; thus, his testimony constitutes lay opinion regarding this specialized subject matter. He did not present any competent substantial evidence to support his contention that the PP exam does not meet generally accepted testing standards.14 Petitioner also testified that item response theory, which is a psychometric tool used in developing and scoring the PP exam, is an invalid means of determining the competence of an engineer. To this point, Petitioner testified that the "model law engineer" is not a real person, but is instead an imaginary person created by subject matter experts to define what a minimally competent engineer should know. Thus, according to Petitioner, 14 Because Petitioner was not qualified, tendered, or accepted as an expert in these specialized subject matters, and because his testimony on these matters consists of inadmissible lay opinion testimony, this testimony has not been afforded weight. §§ 90.701 and 90.702, Fla. Stat. the model law engineer standard is the wrong reference point for determining minimal competency to practice engineering.15 Petitioner also testified that the PP exam does not accurately measure ability, which is a latent trait for which an arbitrary measurement scale must be created. He testified that the model law engineer standard is the midpoint of this scale, and that the purpose of the scale is to determine whether examinees fall above or below that midpoint.16 He further contended that the PP exam does not accurately measure ability, because performance on the exam may be influenced by extraneous variables, such as test anxiety. Petitioner also testified regarding item response theory, which, as previously noted, is a psychometric tool used in developing and scoring high- stakes exams—a subject about which Petitioner had no training in, or knowledge about, until he prepared for the final hearing in this proceeding. Specifically, Petitioner testified that the item characteristic curve is the basic building block of item response theory, and that there are two technical properties of an item characteristic curve: difficulty of the item, and the ability of the item to discriminate between examinees' abilities. Petitioner testified that another basic principle of item response theory is that the examinee's ability is a variable with respect to the items used to determine it. According to Petitioner, this principle rests on two 15 Petitioner's testimony on this point was based on excerpts from a book titled Item Response Theory and a book titled The End of Average. The Item Response Theory book is a treatise on psychometrics, a highly specialized field about which Petitioner was not qualified to testify as an expert, and which is not susceptible to lay witness opinion testimony. Thus, Petitioner's testimony on these points is not assigned weight. See §§ 90.701 and 90.702, Fla. Stat. Additionally, excerpts from The End of Average were determined irrelevant, so were not admitted into evidence. 16 As support for this testimony, Petitioner selectively cited and quoted the deposition testimony of Timothy Miller, Respondent's expert on the development and scoring of NCEES's psychometric-based PP exams. The specific context of Miller's deposition testimony was that when an exam item is overexposed, it is subject to drift, which means that the percentage of correct answers for the item increases to the point that the item no longer is a good discriminator. As further discussed below, Petitioner's testimony on this point was directly and persuasively countered by Miller's expert testimony regarding scaled scores and setting the passing score for the PP exam. conditions: that all items measure the same underlying latent trait, and the values of all item parameters are in a common metric. According to Petitioner, this principle reflects that the item characteristic curve spans the entire ability scale; thus, the practical implication is that a test located anywhere along the ability scale can be used to estimate an examinee's ability, such that an examinee could take a test that is easy or hard, and on average, would score at the same estimated ability level. Petitioner testified that this stands in contrast to classical test theory, which he contends is a better discriminator of examinee ability.17 Petitioner also testified that the psychometric methods used to develop and score the computer-based PP exams are flawed because "difficulty" is subjective and entirely dependent on the individuals developing the PP exam. Thus, according to Petitioner, in scoring a computer-based multiple-choice PP exam, it is impossible to know whether a particular examinee got the answer right due to a reasonable approach in answering the question, or by guessing. Petitioner contends that for this reason, multiple-choice test questions developed using item response theory are not good discriminators of examinees' ability; thus, even if an examinee does not correctly answer enough questions to pass the exam, that does not mean that the examinee is not knowledgeable in that area.18 Petitioner further testified that because difficulty is a subjective parameter, different forms of the PP exam inherently have different levels of difficulty. Thus, according to Petitioner, it is a matter of luck whether an examinee takes a more difficult form or an easier form of the exam. Further to this point, Petitioner testified that because an examinee does not take multiple forms of the exam, but instead takes only one form, the determination of the examinee's ability is solely dependent on a subjective 17 Refer to note 15, supra. 18 Refer to note 15, supra. parameter—i.e., the difficulty of the test questions as determined by subject matter experts. Petitioner contends that, as a result, the PP exam does not accurately measure an examinee's ability, and, therefore, is not a valid exam.19 Petitioner also testified that because statistical indices of reliability and validity are not attributes of an exam, a researcher may select what seems to be an appropriate test for his or her purposes, when, in fact, the selected test does not have any level of reliability or validity. Thus, Petitioner testified, reliability and validity are values that reside in test scoring, not in the test itself. Petitioner testified that validation, in statistics, is the process of accumulating evidence that supports the appropriateness of the inferences that are made of student responses for assessment uses. He testified that validity refers to the degree to which the evidence indicates these interpretations are correct and the manner in which the interpretations are used is appropriate.20 Petitioner testified regarding three types of validity evidence: content, construct, and criterion evidence.21 Specifically, Petitioner testified that content evidence refers to the extent to which an examinee's responses to a given assessment reflect the examinee's knowledge of the content being tested; thus, to the extent an exam inadvertently measures a parameter that is not related to the examinee's knowledge of the content being tested, it is invalid. 19 Refer to note 15, supra. 20 Petitioner's testimony relied on, or was paraphrased from, a document titled The Scoring Rubric Development. Again, because this topic and document address a matter within the specialized fields of psychometrics, high-stakes testing, and test-scoring statistics, which are areas in which Petitioner was not qualified as an expert, and which are not susceptible to lay opinion testimony, pursuant to sections 90.701 and 90.702, Petitioner's testimony relying on this document, including his testimony regarding content, construct, and criterion-related evidence, is not afforded weight. 21 Refer to note 15, supra. This determination regarding the weight afforded Petitioner's testimony applies to paragraphs 94 through 100 herein. According to Petitioner, the content-related evidence for the PP exam for each discipline is inconsistent, so that the PP exam for a given discipline does not accurately measure minimal competence for that discipline. Petitioner also testified that the weighting of different topics on the PP exam necessarily creates an advantage for engineers who work in areas more heavily weighted on the exam, while creating a disadvantage for engineers who work in areas that are less heavily weighted on the exam. Petitioner also testified that to accurately determine minimal competence in all engineers, the model law engineer standard should be keyed to, and the content tested on the exam should be directed toward all engineers, including those having more than four years of experience. Petitioner also testified that construct-related evidence consists of external benchmarks, such as results and explanations, of internal evidence of psychological processes, such as reasoning. Petitioner testified that because multiple-choice exams do not provide evidence of an individual's reasoning process, they do not generate construct-related evidence for purposes of determining exam validity. Petitioner testified that free response paper-and pencil-exams provide construct-related evidence, so are better indicators of an examinee's knowledge. Petitioner also testified regarding criterion-related evidence, which relates to the extent to which the results of an assessment, such as the PP exam, correlate with a current or future event. By way of illustration, Petitioner testified that criterion-related evidence considers the extent to which a student's performance may be generalized to other relevant areas. Petitioner testified that an examinee's performance on the PP exam is not generalizable to other relevant activities, so it is impossible to determine whether the exam actually corresponds to minimal competence in the workplace. In sum, Petitioner testified that the PP exam does not meet content, construct, or criterion-based evidence for purposes of determining whether it is a valid exam. Thus, Petitioner reasons, it is logical to conclude that because the PP exam is not a good discriminator between minimally competent and incompetent engineers, it does not reliably and adequately measure competence. Petitioner also testified that because passing the PP exam is only one component of licensure, it fails to meet criterion-based validity, in that the exam, by itself, does not certify a passing examinee to practice as a P.E. As Petitioner put it, "you're just passing the exam as part of the requirement for licensure." Petitioner reasoned that if passing the PP exam corresponds to minimal competence, the experience and education requirements of section 471.015(2)(a) are redundant. Also to this point, Petitioner testified that the preapproval process to take the PP exam is directly related to an examinee's actual work experience as an engineer, while taking the exam merely entails answering questions about engineering work. Thus, Petitioner contends, a competent engineer, as shown through Respondent's preapproval process, may nonetheless fail the exam. Petitioner asserts that this further shows that the exam does not accurately measure minimal competence. Petitioner also testified that, in his view, delaying licensure of potentially competent engineers due to postponing the exam due to the Covid-19 pandemic does not serve the interest of public health and safety. To that point, he testified that the inability to obtain a variance, which would relieve examinees from having to take and pass the exam under such circumstances, renders the Challenged Rule arbitrary.22 Petitioner also contended that passing the PP exam should not be required, because other engineering professional associations—specifically, the European Federation of National Engineering Associations ("FEANI"), which represents engineers in European countries—allow licensure through 22 Petitioner's point regarding inability to obtain a variance or waiver is addressed in the Conclusions of Law. education and experience requirement, without requiring a professional exam to be taken and passed. However, because section 471.015, which governs the licensure of engineers in the state of Florida, requires a professional licensure examination to be taken and passed as part of the P.E. licensure requirements, Petitioner's testimony and argument regarding FEANI's practices and requirements take issue with the statute, rather than the Challenged Rule, and, thus are irrelevant. Petitioner also contends that the examination fee for the PP exam is arbitrarily set, rendering the Challenged Rule arbitrary. However, as discussed above, the Challenged Rule only addresses determining the passing grade for the PP exam using psychometric methods. The Challenged Rule has nothing to do with establishing or setting an examination fee. Thus, this challenge ground has no basis in fact or law.23 C. Findings Regarding the Evidence Presented in Respondent's Case Respondent's Expert Witnesses Respondent presented the testimony of Timothy Miller, P.E., who serves as Director of Examination Services for NCEES. Miller has held this position for approximately 15 years. His job-related activities and responsibilities include directing exam development, publication, scoring, and fulfillment of the licensing exams for engineers and surveyors; coordinating exam development committees consisting of over 800 volunteer subject matter experts who work on developing each NCEES exam; overseeing the exam development process and providing advice and guidance regarding engineering exam development, administration, production, scoring, analysis, and reporting; serving as a testing process consultant to exam development committees; and other exam-development and administration-related matters. 23 Additionally, this challenge ground was not raised in the Rule Challenge Petition, so is not at issue in this proceeding. See § 120.56(1)(b), Fla. Stat. Before Miller was promoted to his current position, he served as an examination development engineer for NCEES. In that position, Miller was responsible for planning and coordinating engineering exam development, production, administration, scoring analysis, and reporting for certain assigned examinations; serving as a testing consultant working with engineering exam development committee chairs regarding quality and number of exam development volunteers; and overseeing development and administration of the licensing exams in the specific fields of environmental controls systems, metallurgical engineering, and mechanical engineering. Through his experience in these positions with NCEES, Miller is an expert in professional examination development and scoring, particularly with respect to the development and scoring of the NCEES FE and PP examinations. Before being employed with NCEES, Miller practiced civil and structural engineering with several private-sector engineering firms. He has been a professional engineer since 1984, and is licensed as a P.E. in South Carolina, North Carolina, Maryland, Delaware, Pennsylvania, and New Jersey. Respondent also presented the testimony of Dr. Michelle Rodenberry, P.E., an associate dean and associate professor at the Florida A&M University–Florida State University College of Engineering. Her engineering expertise is in the field of structural engineering—specifically, bridge engineering. Rodenberry was appointed to the Board in 2012, and she served as a Board member until 2018. She is now an emeritus Board member. While on the Board, she served as chair of the education committee, and was involved in reviewing applications for licensure as a P.E. in Florida. Development, Scoring, and Validation of the PP Exam The NCEES engineering exams are national licensing exams that are recognized by every engineering licensing entity in each of the U.S. states, as well as by the engineering licensing entities in Washington, D.C.; Puerto Rico; the U.S. Virgin Islands; and the other U.S. territories and protectorates. There are approximately 26 different engineering disciplines, each of which is tested by a separate PP exam specific to that discipline. In the 1990s, NCEES decided to transition from subjectively-graded pencil-and-paper examinations to an objectively-graded computer-based multiple-choice exam format. Currently, approximately one-third of the PP exams, including environmental engineering, have been converted to a computer-based format, and all but one of the exams in the other disciplines is in the process of being converted. The reason NCEES is transitioning the PP exam from a pencil-and- paper format to a computer-based format consisting of multiple-choice questions is to help eliminate subjectivity in grading, so that the exam papers are consistently graded across groups of examinees. Additionally, a computer-based format consisting of objective multiple-choice questions allows the difficulty of the exam to be psychometrically evaluated for purposes of determining the passing score for a particular administration of the exam. To that point, because computer- based multiple-choice exams are objectively scored, exams offered at different times during the year are able to be compared, or equated, for purposes of setting the passing grade for a particular exam administration.24 Respondent entered into a contract with NCEES in 2009, pursuant to which NCEES provides the FE and PP exams for engineer licensure in Florida. In 2013, the contract was amended to allow NCEES to provide the exams by computer-based testing, using Pearson Vue as its exam 24 As Miller explained,"[i]f they were different on a difficulty level, the harder exam, the standard would actually be lowered so that it would be fair across administration so everybody was treated consistently. Or if my exam was less difficult, the standard would be raised. I would have to get more questions right." administering entity. The FE and PP exams are administered by Pearson Vue at its testing centers. NCEES develops model laws and rules that represent best practices with respect to state licensing of engineers. The aim of these model laws and rules is to achieve uniformity and consistency throughout the states and the U.S. territories and protectorates in the licensure of professional engineers. A significant benefit of such consistency and uniformity is the resulting "mobility" for licensed professional engineers—that is, the ability to more easily become licensed to practice engineering in multiple states. The NCEES model laws and rules establish the "model law engineer," which defines and constitutes the standard for minimal competence in a specific engineering discipline for purposes of being licensed as a P.E. in that discipline. The model laws and rules define the "model law engineer" as a person who holds a degree from an engineering educational program accredited by ABET, has four years of active engineering practice experience, and passes the FE and PP exams. The model law engineer standard equates to the competence level of an engineer having four years of engineering experience who is capable of practicing engineering in a manner that protects the public health and safety. This constitutes the minimum competence level that an applicant must demonstrate for purposes of being licensed as a P.E.25 in the 50 states and the U.S. territories and protectorates. Thus, the NCEES PP exam is constructed to test engineering ability keyed to the model law engineer standard. That is, the PP exam is designed to determine the ability level of an applicant for P.E. licensure for purposes of 25 Refer to note 12, supra. The term "minimal competence," as used in the Challenged Rule, is specifically keyed to the "model law engineer" standard for purposes of being licensed as a P.E. It is not meant to indicate or imply that an engineer who does not take or pass the PP exam is per se incompetent, such that he or she is not competent to engage in work constituting engineering, as defined in section 471.005(7). comparing that ability level to that of an engineer having four years' experience who is able to practice engineering in a manner that protects the public health and safety. Examinees having four years of engineering experience after graduation have the highest pass rates on the PP exams. Pass rates for examinees with more or less than four years of experience are lower, typically in proportion to the length of time before or after the four-year experience mark when they take the PP exam. Miller explained, credibly and persuasively, that the reason for the drop-off in PP exam performance after the four-year mark is that "life happens." Engineers gain more experience, and many become specialized in a relatively narrow niche, or move into managerial, non-technical positions. Additionally, because the PP exam does, in part, test subjects that one learns in college, the longer an examinee is out of college, the less subject matter recall in certain areas he or she may have. "Psychometrics" is the specialized field of study concerned with the theory and technique of psychological measurement. Specifically, psychometrics entails the objective measurement of skills and knowledge, abilities, and educational achievement. Among other specialized areas of practice, psychometricians focus on the construction and validation of assessment instruments, and theories, such as item response theory, that relate to psychological measurement. Psychometricians typically have graduate training and all possess specialized qualifications that enable them to engage in objective psychological measurement. PP exams are designed to determine minimal competence in a specific engineering discipline. "Minimal competence" is the minimal amount of knowledge required to practice in that particular engineering discipline in order to protect the public health and safety. For any specific engineering discipline for which it has been determined that a PP exam should be given,26 there is an approximately three-year due diligence period in which subject matter experts in that discipline work to determine the topics that should be tested on the exam. The PP exam for each specific discipline is developed by subject matter experts, who volunteer and meet on a monthly basis to develop, review, and evaluate the questions for the PP exam for that specific discipline.27 The process of determining which topics should be tested on a PP exam, termed the Professional Activities and Knowledge Study ("PAKS") process, is a standard practice used to determine the specific topics to be tested on a PP exam. As part of the PAKS process, a consulting psychometrician28 employed by Pearson Vue; 15 to 20 engineers who are licensed in another engineering discipline; and subject matter experts who may teach a particular engineering discipline for which the PP exam is being developed, work together to develop consensus regarding the specific topics that engineers having four years of experience practicing in that discipline need to know in order to safely practice engineering in a manner that protects the public health and safety.29 26 For a PP exam to be developed for a new engineering discipline, at least ten NCEES- member state engineering boards must request that such exam be developed, and at least one ABET-accredited program in that specific discipline must exist. 27 Over the years of development and administration of the PP exam, hundreds of licensed engineers have provided input regarding the topics that should be, and are, tested in each discipline and the relative weight given to each topic on the PP exam. 28 Pearson Vue's psychometricians who develop, score, and evaluate NCEES's exams have Ph.D. degrees in psychometrics or statistical analysis. 29 Subject matter experts selected to develop the PP exam questions are chosen based on consideration of the type of practice, such as governmental and private practice; gender; ethnicity; length of time of licensure as a P.E.; and geographic considerations. All subject matter experts must be licensed as a P.E. by a state engineering licensure board in order to The consulting psychometrician builds a questionnaire that lists the specific topics identified by the PAKS committee, and distributes an online survey to engineers who practice in the discipline for which the PP exam is being developed. The survey seeks input regarding the relative importance of each specific topic for purposes of testing to demonstrate minimal competence in the discipline. Based on the survey responses from engineers practicing in the discipline, exam specifications are developed. The exam specifications identify each specific topic to be tested on the PP exam, and the number or percentage of exam questions that will address each specific topic within that discipline. The exam specifications must be approved by an oversight committee. Once the exam specifications have been approved, the subject matter experts for that specific engineering discipline for which the PP exam is being developed prepare the PP exam questions—also termed "items"—and review and evaluate them for clarity, demographic neutrality, and other parameters, so that the items will reliably and validly test engineering ability. In computer-based multiple-choice PP exams, the questions are prepared such that for each question, there is only one correct answer and three other plausible, but incorrect, alternative choices. The individual exam questions are reviewed numerous times by the subject matter experts before they are moved into an exam question bank for use on the PP exam. Once the exam questions have been developed and banked for use on a PP exam, a standard-setting committee, consisting of ten to 15 licensed engineers having diverse backgrounds, experience, and demographic features, reviews the exam to determine the minimum passing score—or "cut score"—on the exam. The cut score equates to the ability level of an engineer serve in this capacity. As noted above, over the years of development and administration of the PP exam, hundreds of engineers have provided input in developing each PP exam. having four years of experience who is minimally competent to practice engineering at a level that protects the public health and safety. This method of using subject matter experts to examine the content of each exam question and predict how many minimally-qualified examinees would answer each question correctly is termed the "Modified Angoff Method." The standard-setting committee then takes the exam, and the psychometrician analyzes the data from the committee's exam sitting. Using this data and analysis, the standard-setting committee then reviews, and reaches consensus, regarding each question, for purposes of determining the proportion of minimally competent engineers who would answer that specific question correctly. Based on the information generated by this process, the psychometrician develops the "panel recommended passing score," with a statistical margin of error. The psychometrician presents this recommended passing score to a committee of five persons, consisting of two state licensing board members and three subject matter experts who observed the exam development process. Based on the psychometrician's recommendation, the committee makes the final decision regarding the minimum passing score for the exam. Each PP exam question is developed and evaluated using the process described above, and is placed in bank for use on a PP exam. The psychometrician uses item response theory to "calibrate"—i.e., determine the relative difficulty level of—each exam question.30 An exam question is not banked for use on future sittings of the exam unless it has had at least 200 responses on a previous exam, so that statistics for each item's performance can be generated for purposes of item calibration. 30 Item response theory is one of many psychometric methods, or tools, used to weight exam questions for purposes of creating different forms of exams having the same level of difficulty. Depending on the specific discipline, a question bank for a PP exam may consist of many thousands of questions.31 Using item response theory, the psychometrician converts the passing score to create a scale from -5 to +5, which will equate to the examinee's ability level as measured by the exam. Once the passing score for the PP exam has been determined, different PP exam "forms" are created for administration in different exam sittings. Exam "forms" are essentially different versions of the PP exam that consist of different individual questions of the same difficulty level, as determined using item response theory, for each specific topic on the exam. Thus, if a PP exam was administered, for example, in April and October, the different exam administrations would consist of different forms—meaning that the exam would consist of different questions, but the questions would be of the same difficulty level for a specific topic tested on the exam.32 Additionally, because exam item difficulty has been calibrated using item response theory, different forms of a PP exam can be given during the same exam administration at different locations.33 Importantly, because the difficulty of the exam items has been calibrated using item response theory, the different exam forms are statistically equivalent in difficulty. 31 The only items that will be used on the graded portion of the PP exam are questions that have known statistics such that they have been calibrated for difficulty. However, there may be other "pretest" questions on the exam strictly for purposes of gathering statistics regarding performance on the questions for potential inclusion in the exam item bank; these "pretest" exam questions are not graded for purposes of determining the examinee's score on the exam. 32 As Miller explained, for an administration of an exam at different locations at the same time, the form administered at a particular location consists of different questions than the form administered at another location; however, the exam forms are equivalent in terms of the number of questions addressing a particular topic and the difficulty of the items addressing that topic. 33 Using the "linear-on-the-fly" ("LOFT") method to generate different forms of the exam also helps ensures exam security, since persons sitting near each other during an administration of the exam will not have the same exam form. As a result of using these processes, including the Modified Angoff Method, and applying item response theory to calibrate the exam items for purposes of constructing different, but statistically equivalent, forms of the PP exam, examinees are not graded on how they perform against each other, but instead are graded against the cut score set for the exam. To ensure that different forms of the exam are statistically equivalent in difficulty, Pearson Vue uses the LOFT method,34 which employs an algorithm to ensure that, across all of the exam forms, all examinees get the same number of questions of the same level of difficulty on the same topics. The algorithm randomly generates, or assembles from banked exam questions, different exam forms based on the exam specifications (i.e., the specific topics tested and relative weight/number of exam questions for that topic) and the difficulty level of the exam questions, such that the different exam forms generated by LOFT are statistically equivalent to each other. Using item response theory to calibrate specific exam question difficulty based on the statistical probability of being answered correctly enables examinees taking different, but statistically equivalent, forms of the exam to be compared to the passing standard for purposes of determining whether they pass the exam. Thus, examinees are compared to an ability level—here, minimal competence—rather than to each other.35 This method ensures that all examinees take an exam of equivalent difficulty, which, in turn, helps ensure the fairness of the exam. 34 For the engineering disciplines having too small a group of examinees to employ item response theory or LOFT to generate different exam forms, each examinee takes the same exam instead of taking different forms of the exam, and the exam typically is offered only on one day, rather than multiple days, per year. 35 By way of example, Miller explained that if two examinees each answer five questions on the same topic on the exam, and one examinee answers four easier questions correctly and the other examinee answers two comparatively more difficult questions correctly, the examinee answering the two more difficult questions correctly may have a higher ability level on that particular topic, due to the comparative difficulty of the questions that examinee answered correctly. Once a PP exam is administered, Pearson Vue scores the examination and sends NCEES the information regarding whether each examinee has passed or failed the exam. Pearson Vue also provides each examinee the information regarding his or her performance on the exam compared to the minimum competence standard. The examinee's performance is expressed as a scaled score, for each specific topic tested on the exam, and for the entire exam. Specifically, using psychometric statistical methods, the ability level of the examinee is expressed as "theta," and is placed on a scale of 0 to 15 for each of the specific topics tested. The examinee's overall theta across all specific topics tested is then compared to the "minimal competence" passing standard, which is also expressed as a scaled score using the same 0 to 15 scale. After an exam is taken and scored, the consulting psychometrician analyzes this data, called "response data," for each exam question, for each examinee, to calibrate the items for purposes of determining whether a particular question performs well in discriminating ability level of the examinees. The psychometrician may recommend that an exam question be "retired" because it is not performing as a good discriminator of ability level. Examples are where an exam question is too difficult or too easy, such that it does not discriminate well in determining ability level; where an item takes too long to answer or is ambiguous; where an item has been "overexposed" by having become publicized such that future examinees have access to the question and scores on the question become high; or where an exam contains "bad pair" items, such that the answer to one item may suggest, or lead to, the answer on another similar item. Having a psychometrician involved in tracking and analyzing exam data enables such circumstances and situations—which may influence the scores on a test item for reasons not related to the examinee's ability—to be identified and corrected. Returning to a free response, paper-and-pencil exam format for the P.E. licensing exam would provide a far less objective, fair, and accurate measure for determining minimal competence for purposes of being licensed to practice engineering. It also would negatively affect the ability of licensed engineers to become licensed in other states. Due to the use of psychometric methods in developing and scoring, the PP exams are very reliable across multiple administrations of the exam— to the point that all of NCEES's psychometric-based PP exams score upwards of .9 on a scale of 1.0.36 Psychometric methods, including item response theory, are used in developing, administering, and scoring many different types of high-stakes professional and academic examinations, including medical school admissions examinations, and nursing, medical examiner, internal auditor, and architecture licensure examinations. NCEES audits approximately one-third of the exams administered by Pearson Vue on an annual basis, to independently evaluate the accuracy of the psychometric services provided by Pearson Vue, and to ensure that the exams have been created pursuant to NCEES's guidelines, procedures, and requirements. NCEES also retains independent psychometricians to review Pearson Vue's exam-related reports and analyses, to ensure that Pearson Vue is following standard psychometric rules of good practice. In sum, the use of objective psychometric methods, including the methods discussed above, to develop, score, and evaluate the PP exam ensures that minimal competence, for purposes of licensure as a P.E., is accurately measured by the exam. Role of the PP Exam in Licensure of PEs in Florida As discussed above, to be licensed as a P.E. in Florida, an applicant must have a college degree from an ABET-accredited institution, four years of 36 Test reliability refers to the degree of consistency with which a test measures a particular subject matter across different administrations of the test. A test has a high reliability score if it consistently produces similar results under consistent conditions. A 1.0 reliability score reflects perfect consistency in results across different administrations of a test. An acceptable reliability score target for high-stakes tests is .7 or higher. active experience in engineering practice, and have passed the FE and PP exams. Thus, the P.E. exam is a vital component of determining that an engineer licensed as a P.E. to practice in Florida is able to practice at a competence level that protects the public health and safety. Unlike the education and experience requirements for licensure— both of which may entail a great deal of variability in quality and breadth across applicants—the PP exam constitutes an objective, consistent tool37 to measure an applicant's level of competence for purposes of determining whether the applicant possesses the minimal competence needed to practice engineering in a manner that protects the public health and safety. As such, the PP exam constitutes a uniform measure of minimal competency for purposes of licensure as a P.E. in Florida. As discussed above, this does not mean that a person who engages in engineering work but has not passed the PP exam is incompetent; it merely means that he or she has not demonstrated minimal competency on this required objective measure of competency for licensure purposes. As discussed above, the PP exam is specifically designed to ensure that a licensed P.E. is competent to practice over a range of specific areas encompassed within a particular discipline. This is because a licensed P.E. is authorized to practice engineering within any discipline or area, subject to professional and ethical standards. The breadth of the PP exam thus helps ensure minimal competence to practice engineering in a manner that protects the public health and safety. 37 This consistency and uniformity is the direct result of the psychometrically-based exam development, scoring, calibration, and validation methods discussed above. The purpose of P.E. licensure is to inform and protect the public, which is entitled to rely on such licensure as indicating that the licensee is competent to practice engineering.38 Administration of the PP Exam During the Covid-19 Pandemic As discussed above, Petitioner has alleged that the Challenged Rule is arbitrary on the basis that it does not address contingencies for offering the exam if unforeseen circumstances prevent regular administration of the PP exam. Specifically, Petitioner points to the fact that Pearson Vue cancelled the April 2020 PP exam administration due to the Covid-19 pandemic. Pearson Vue has rescheduled the PP exams for various times and at various locations around the country in an effort to make the PP exam available for prospective examinees during the pandemic.39 Pearson Vue also is taking substantial steps to protect persons who have applied to take the PP exams during the Covid-19 pandemic. To that point, Pearson Vue has retrofitted its testing centers to help ensure the safety of the examinees as they sit for the PP exam. Specifically, the number of examinees who will be in a testing room has been reduced; masks are required to be worn by examinees and proctors; testing stations are cleaned between each use; some additional test center locations have been added; and some states have relaxed rules to allow examinees to sit for the exam in states other than the one for which they are applying for licensure. At present, the exams are not able to be offered over the internet so that examinees are able to take the exam at a remote location. A substantial reason for this is lack of exam security, which is necessary to protect and 38 As noted above, a person does not have to be licensed as a P.E. to engage in engineering work in Florida. However, if a person wishes to hold himself or herself out to the public as a P.E., then that person must satisfy the requirement to pass the PP exam, which is an indicator of minimal competence for purposes of licensure. 39 Among other things, Petitioner alleges, in paragraph 19 of the Rule Challenge Petition, that the Challenged Rule is invalid under section 120.52(8)(d) because it does not address circumstances where an examination cannot be administered due to force majeure. As maintain the exam's integrity. Additionally, the lack of guarantee of internet service reliability and functionality for every examinee is a crucial consideration, since failed internet connections could significantly affect the fairness of the exam.
The Issue The issue presented is whether Respondent And Justice For All, Inc., d/b/a Legal Club of America is a legal expense insurance corporation, and, if so, whether it is exempt from regulation by the Department because it is authorized by The Florida Bar.
Findings Of Fact Respondent is a Florida corporation doing business in Florida and nationwide. In 1996 Legal Club of America was purchased by And Justice For All, Inc. The company continued doing business as Legal Club of America, offering a lawyer referral service that has remained the same since its inception. In 1998 And Justice For All, Inc., was acquired by a company that became Legal Club of America Corporation. And Justice For All, Inc., d/b/a/ Legal Club of America is a subsidiary of Legal Club of America Corporation. Respondent is a private lawyer referral service structured as a club membership plan. Respondent operates as a purchasing cooperative, in which the combined purchasing power of its members is connected to attorneys willing to offer free and discounted services in exchange for new clients. Respondent offers two membership plans in Florida: a Family Plan and a Small Business Plan. The basics of the plan are simple: a club member pays an annual membership fee and receives a referral from Respondent to a plan attorney who has agreed to abide by a schedule of free and discounted services. There have been no substantive changes in the structure of Respondent's membership plan since its inception. The only change to the plan has been an increase in the amount of the membership fee. When a plan member signs up with Respondent, the member is assigned a particular attorney. When a legal issue arises, the member contacts the attorney directly. If Respondent is unable to refer a plan member to a participating attorney who will abide by the fee schedule outlined in the plan member guidebook, the plan member is entitled to a refund of the membership fee paid to Respondent. Respondent does not indemnify or reimburse any plan member for any attorney's fees. Respondent does not indemnify or reimburse any plan attorney for any attorney's fees. Club members are responsible to the plan attorneys for all legal fees incurred. The plan member must contract directly with the attorney for legal services and is responsible for payment of all legal fees incurred. The membership fees which Respondent charges plan members are fixed at a predetermined rate that is not affected by any particular legal problems of a plan member or the frequency with which the plan member uses an attorney. Plan attorneys may charge plan members fees far in excess of the membership fee paid to Respondent. For example, if a plan member needs legal representation exceeding the free or discounted services included in the plan, the plan member and the attorney are free to negotiate representation for a mutually agreeable amount. Respondent is not engaged in the practice of law. Respondent does not perform legal work for plan members nor does Respondent provide any substantive assessment of the plan member's legal situation or need for legal assistance. The plan attorneys are solely responsible for providing legal services to plan members. Plan attorneys are not employees of Respondent and have no financial obligation to Respondent. Plan attorneys do not pay anything to Respondent. Respondent has no attorneys on its staff that do legal work for plan members. All legal work is performed by private attorneys who do not receive any compensation from Respondent. Respondent does not pay participating attorneys any fee for accepting referrals or for participating in the plan. The only thing Respondent provides to plan members is a referral to a plan attorney. The fee schedule arranged between Respondent and participating attorneys applies only if the attorney accepts the referral. Plan attorneys are free to refuse a referral from Respondent for any reason other than Respondent's rates. If a refusal occurs, Respondent refers the plan member to another attorney on the plan. Participating attorneys must be licensed in Florida and must maintain at least $100,000 per incident and $300,000 aggregate professional liability coverage. Respondent's plans do not have any limitations on usage or waiting periods, and they do not exclude pre-existing legal problems. Respondent does not become involved in the confidential attorney/client relationship. The plan member guidebook provides that all legal matters are guaranteed to be exclusively between the plan member and the plan attorney. The plan member guidebook further provides that Respondent is not privy to and does not have access to client files or records and does not interfere with the professional independent judgment of plan attorneys. The participating attorney agreement informs plan attorneys that Respondent is not an insurance product or pre- paid legal service of any kind. Respondent's plan member guidebooks similarly inform members that Respondent is not a law firm, an insurance carrier, or a provider of legal services. Before entering the Florida market, Respondent contacted the Department by telephone to inquire whether Respondent would be subject to Chapter 642, Florida Statutes, (legal expense insurance) regulation. In a letter confirming the telephone call, Respondent asked the Department for written confirmation of the Department's determination that Respondent is not a pre-paid legal plan and is not subject to regulation by the Department. In response to Respondent's request for written confirmation, the Department notified Respondent by letter dated March 25, 1994, that the plan, as structured, was a referral service and would not be subject to regulation under Chapter 642. The Department did recommend that Respondent file its package with The Florida Bar to determine whether Respondent is a lawyer referral service. In response to that recommendation, Respondent contacted The Florida Bar on March 28, 1994, and in response to a telephone request from the Bar, sent written information to the Bar on March 29. The Bar provided copies of these materials to the Department. After reviewing the materials, the Department advised Respondent that it had changed its mind and Respondent would have to apply for a license from the Department. On April 20, 1994, Respondent faxed a letter to the Department requesting reconsideration of that determination and a retraction of its decision. On that same day, Respondent had a telephone conversation with the Department, at the conclusion of which the Department re-affirmed its earlier determination that Respondent is a lawyer referral service and not subject to regulation by the Department. On April 21 Respondent confirmed this re-affirmation by letter to the Department and requested a written acknowledgement. On April 27 the Department confirmed the Department's determination that Respondent is not subject to regulation under Chapter 642, Florida Statutes, and is not a pre-paid legal expense corporation. In response to Respondent's request for an opinion from The Florida Bar regarding whether Respondent is subject to any regulations of The Florida Bar, the Bar, on April 13, 1994, advised Respondent that Respondent is a private lawyer referral service subject to the reporting requirements of Bar Rule 4-7.8. Since 1994, Respondent has filed quarterly reports with The Florida Bar in accordance with the requirements of that Rule. The current position of an employee of The Florida Bar is that Respondent is not a lawyer referral service authorized by The Florida Bar. The Bar removed Respondent from its list of private referral companies only after the Department advised the Bar in 1998 of the Department's current claim that Respondent is subject to regulation by the Department as a provider of legal expense insurance. There is no Bar rule that prohibits Respondent from operating in Florida and no Bar rule that prohibits any Florida lawyer from accepting referrals from Respondent. Rather, Rule 4-7.8(b)(1) defines lawyer referral services and covers the manner in which Respondent operates, notwithstanding the testimony of a Bar employee that the intent of the Rule is different from its plain language. The Rule, as written, was approved by the appropriate Bar committee, by the Board of Governors of The Florida Bar, and by the Supreme Court of Florida. The Rule contains no intent statement that differs from its plain language. All legal expense insurance companies licensed in Florida share a common characteristic that distinguishes them from Respondent: they all either reimburse policyholders for legal expenses incurred or pay attorneys to provide legal services to policyholders. Respondent does neither. In 1996 a pre-paid legal expense insurance company regulated by the Department contacted the Department and complained that Respondent was not being regulated. This prompted the Department to send a letter to Respondent telling Respondent to cease and desist operating immediately. The Department did not conduct any independent investigation into Respondent's business operations before sending the cease and desist letter. The letter was sent based solely upon the complaint made to the Department and a news article about Respondent sent to the Department by the complainant. Upon receipt of the "cease and desist" letter, Respondent wrote to the Department reminding the Department of its previous determination that Respondent is not subject to regulation by the Department and stating that Respondent's method of doing business had not changed since that determination was made. This letter was followed by a telephone conversation on December 16, 1996, which conversation was confirmed in a December 16 letter from Respondent to the Department and in a December 17 Department letter. In its confirming letter, Respondent enclosed current materials showing the company's relationships with its members and the attorneys on its referral panel. Respondent reiterated that Respondent's structure had not changed in any respect since 1994, other than an increase in the amount of the membership fee, which is an administrative fee wholly retained by Respondent for the costs of the membership materials and administration of the referrals. In an internal E-mail message on December 17 the Department confirmed that Respondent's membership plan was determined to be exempt from regulation by the Department in 1994 and that Respondent had since that time been filing quarterly reports with The Florida Bar. By letter dated February 21, 1997, the Department advised Respondent that based upon the Department's review of Respondent's plans it appeared that Respondent was acting as a lawyer referral service. The Department expressed some concern, however, about the cost of the membership fee Respondent charges its members. Respondent responded to the Department's letter on March 5, 1997, explaining that Respondent's memberships are sold at many different price levels, including bundled prices sold through associations for as little as $2.10 per year per member. On April 21, 1997, stating, without further explanation, only that Respondent's plan had changed since 1994, the Department sent Respondent a letter declaring Respondent subject to regulation by the Department and directing Respondent to cease doing business until it became licensed by the Department. During discovery in this proceeding, the Department admitted that the only thing about Respondent's plans that had changed between 1994 and 1997 was the amount of Respondent's membership fees. It can be inferred from an internal Department E-mail authored in November 1997, several months after the Department directed Respondent to cease operating in response to the complaint the Department had received, that the Department's action against Respondent is motivated, at least in part, by a desire to please or protect licensed legal expense insurance companies from competition by Respondent. In addition to requesting guidance from Florida regulators before commencing business in Florida, Respondent requested opinions from regulators in other states regarding whether Respondent would be regulated as providing insurance or by the state bar associations. No state determined that Respondent was subject to regulation as providing insurance.
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is RECOMMENDED that a final order be entered determining that Respondent is not selling legal expense insurance; determining that Respondent is not, therefore, subject to regulation by the Department; and withdrawing the Department's Notice of Intent to Issue Cease and Desist Order. DONE AND ENTERED this 3rd day of February, 2000, in Tallahassee, Leon County, Florida. LINDA M. RIGOT Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 SUNCOM 278-9675 Fax Filing (850) 921-6847 www.doah.state.fl.us Filed with the Clerk of the Division of Administrative Hearings this 3rd day of February, 2000. COPIES FURNISHED: Bill Nelson, State Treasurer and Insurance Commissioner Department of Insurance The Capitol, Plaza Level 11 Tallahassee, Florida 32399-0300 Daniel Y. Sumner, General Counsel Department of Insurance The Capitol, Lower Level 26 Tallahassee, Florida 32399-0300 Richard A. Grumberg, Esquire Department of Insurance 200 East Gaines Street 612 Larson Building Tallahassee, Florida 32399-0333 Thomas J. Jones, Esquire Holland & Knight, LLP 315 South Calhoun Street, Suite 600 Tallahassee, Florida 32302 Carroll Webb, Executive Director Joint Administrative Procedures Committee The Holland Building, Room 120 Tallahassee, Florida 32399-1300 Liz Cloud, Chief Bureau of Administrative Code Department of State The Elliott Building Tallahassee, Florida 32399-0250
The Issue At issue in this proceeding is whether Respondent committed the offenses set forth in the Administrative Complaint and, if so, what penalty should be imposed.
Findings Of Fact Respondent, Timothy A. Alexander, is now, and was at all times material hereto, licensed as a physician by the State of Florida, having been issued license number ME 0035285. On June 29, 1995, the Board of Medicine entered a final order which approved and adopted a consent agreement accepted by Respondent in a prior disciplinary action (Case No. 92-11508). Pertinent to this case, the final order imposed an administrative fine in the amount of $1,500.00, against Respondent, which was to be paid within 60 days following the filing of the final order (June 30, 1995). The final order also required that Respondent attend 10 hours of Category I Continuing Medical Education in risk management within one year of the filing of the order. Finally, the order required that Respondent complete the course "Quality Medical Records Keeping for Health Care Professionals," sponsored by the Florida Medical Association, or a Board-approved equivalent, within one year of the filing of the final order.2 Here, there is no dispute or reason to doubt that Respondent timely paid the administrative fine imposed by the final order and that he likewise timely completed the 10 hours of Category I Continuing Medical Education in risk management required by the terms of the final order.3 Consequently, the only viable issue to resolve is whether Respondent timely completed the course "Quality Medical Records Keeping for Health Care Professionals," sponsored by the Florida Medical Association, or a Board-approved equivalent. The course "Quality Medical Records Keeping for Health Care Professionals," sponsored by the Florida Medical Association (the "Course"), is a course designed to help physicians improve their medical record-keeping skills and is divided into two phases. Phase I includes a one-hour credit for preparatory reading and a four-hour credit for on-site instruction in Jacksonville, Florida. The on-site instruction includes one hour of didactic lecture, a two-hour audit practicum, and a one-hour critique of the practitioner's existing records. Phase II is a follow-up critique, designed to be performed at three months following completion of the on-site instruction. At this phase, the practitioner is required to submit another six sets of records for evaluation (the second set), which presumably reflect the benefits of the on-site instruction. The second set of records is evaluated by the same individual who examined the first set and if deemed acceptable the evaluator would immediately notify the Florida Medical Association (FMA), which would issue a certificate reflecting completion of the course. If the practitioner's record-keeping was not acceptable, he would be accorded another three-month period to implement the recommendations, following which he would submit additional records for evaluation. Successful completion of Phase I and Phase II was required for course completion. Here, the proof demonstrates that in or about May 1996, Respondent registered to attend the Course on June 8, 1996, at Jacksonville, Florida, and that on May 9, 1996, the FMA forwarded to him the required preparatory reading material. Respondent attended and completed the June 8, 1996, on-site instruction (Phase I). By letter of August 12, 1996, two months following the on-site instruction, the FMA reminded Respondent that, at three months, he needed to complete Phase II. That letter provided, as follows: The letter is to remind you that it is time for Phase II of the FMA Clinical Excellence Program, "Quality Medical Records Keeping for Health Care Professionals". You completed Phase I on June 8, 1996. The second phase of the course will consist of a self-audit of approximately 10-20 of your own office records utilizing the same audit criteria which were provided in Phase To ensure that the objectives of the course have been accomplished, this audit is performed three months after you have completed Phase I. The three month delay is to allow sufficient time for implementation of the new record-keeping practices in your own medical record keeping system. The same faculty member who critiqued your office medical records during Phase I will provide the critique of your Phase II self- audit. You need not return to Jacksonville to complete the self-audit. At the end of three months, September 8, 1996, six (6) sets of medical records should be mailed to the Florida Medical Association, attention Suzanne Brunette, CME Projects Manager. Please take appropriate measures to preserve patient confidentiality. Your mentor will evaluate the records and report the findings to you. Upon successful completion of Phase II, you will receive a certificate indicating that you have completed the course. No certificate of credit can be issued until Phase II (follow-up evaluation) has been successfully completed. In late October or early November, 1996, Respondent submitted the required medical records for evaluation, they were successfully critiqued, and on November 6, 1996, Respondent was certified by the FMA as having successfully completed the course. Given the proof, it cannot be subject to serious debate that Respondent did not complete the Course, and could not have completed the Course (given the date he elected to take Phase I of the Course and the minimum three-month delay required between the completion of Phase I and the completion of Phase II), within one year after the Board's order was filed. He did, however, timely complete the on-site portion of the Course (Phase I) and successfully completed the follow-up critique (Phase II), with nominal delay. That Phase II and, consequently, completion of the Course occurred outside the one-year period prescribed by the final order was not, apart from the untimeliness itself, shown to be significant.
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is RECOMMENDED that a final order be entered which finds Respondent guilty of violating Section 458.331(1)(x), Florida Statutes, by having failed to timely complete the course "Quality Medical Records Keeping for Health Care Professionals," sponsored by the Florida Medical Association, as alleged in the Administrative Complaint, and that for such violation, Respondent receive a reprimand and an administrative fine in the amount of Fifty Dollars ($50.00). It is further RECOMMENDED that in all other respects, Respondent was not shown to have committed any offense alleged in the Administrative Complaint and that the Administrative Complaint should otherwise be dismissed. DONE AND ENTERED this 12th day of March, 1999, in Tallahassee, Leon County, Florida. WILLIAM J. KENDRICK Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 SUNCOM 278-9675 Fax Filing (850) 921-6847 www.doah.state.fl.us Filed with the Clerk of the Division of Administrative Hearings this 12th day of March, 1999.
The Issue The issue is whether the Amended Order of Penalty Assessment issued to Respondent, Dave's Tractor, LLC, on August 27, 2018, is correct.
Findings Of Fact Respondent is a limited liability company engaged in the construction business with offices at 434 Skinner Boulevard, Suite 105, Dunedin, Florida. It uses tractors and a grading process to prepare land prior to building construction for commercial clients. Its managing member is David Richardson. The Department is the state agency responsible for enforcing the requirement of the Workers' Compensation Law that employers secure the payment of workers' compensation coverage for their employees and corporate officers. § 440.107, Fla. Stat. To enforce this requirement, the Department conducts random inspections of job sites and investigates complaints concerning potential violations of workers' compensation rules. On May 25, 2018, Christina Brigantty, a Department investigator, conducted a routine inspection of a job site at 3691 Tampa Road, Oldsmar, Florida. She observed two men working in a ditch, one man mixing cement, the other man driving a tractor. Investigator Brigantty observed four individuals at the job site, including the two working in the ditch: Dylan Richardson; Ismael Demillon; Javier Mastica; and Jorge Duran. She was informed by the individuals that they worked for Richardson Trailers, LLC. Investigator Brigantty called Mr. Ramsey, corporate officer for Respondent, who confirmed that Respondent hired Richardson Trailers, LLC, as a subcontractor. She later confirmed through discussions with Dylan Richardson and the Coverage and Compliance Automated System that Richardson Trailers, LLC, had no workers' compensation insurance on its employees. The parties have stipulated that at the time of the inspection, Respondent had not secured workers' compensation for any of the four individuals observed on the job site. Investigator Brigantty received approval from her supervisor to issue Respondent a Stop-Work Order and Request for Business Records for Penalty Calculation (BRR). These papers were served on Respondent on June 30, 2018. The BRR requested numerous types of business records for the period May 26, 2016, through May 25, 2018, including business tax receipts (occupational licenses), trade licenses or certifications, and competency cards held by Respondent or any of its principals; payroll documents (time sheets, time cards, attendance records, earnings records, check stubs, and payroll summaries for both individual employees and aggregate payrolls, and federal income tax documents reflecting the amount of remuneration paid or payable to each employee, including cash); and account documents including all business check journals and statements, which would include cleared checks for all open and/or closed business accounts established by the employer. Respondent failed to provide any business records in response to the BRR to determine Respondent's payroll for the audit review period. Therefore, the Department proceeded to compute a penalty based on imputed payroll in accordance with section 440.107(7)(e), Florida Statutes. This formula produced a penalty assessment of $165,654.10. On August 27, 2018, the Department served Respondent with an Amended Order of Penalty Assessment totaling $165,654.10. Pursuant to Florida Administrative Code Rule 69L-6.028(4), the Department also gave Respondent 20 business days in which to provide business records that would confirm Respondent's actual payroll during the two-year review period. This meant the records were due by September 25, 2018. A final hearing was scheduled initially for January 24, 2019. By agreement of the parties, on January 4, 2019, the case was rescheduled to March 15, 2019. One ground for granting a continuance was that the parties were "waiting on outstanding discovery that is being located and is necessary for an amicable resolution," presumably referring to items listed in the BRR. The final hearing was conducted on March 15, 2019, or almost seven months after the Amended Order of Penalty Assessment was issued. A week before the final hearing, Respondent began providing business records to the Department, including bank statements and checks on March 8, 2019, and a general ledger on March 13, 2019. Given the time constraints, they were not reviewed by the auditor until the day before the final hearing. The auditor conceded at hearing that these records would result in a "significantly lower" penalty, and they were sufficient to recalculate the penalty. Even so, at this late date, the Department refuses to recalculate the assessment. Respondent's principal, Mr. Richardson, testified that he has "no way to pay" the penalty, it will force him out of business, and he will be required to terminate his employees. Mr. Richardson also testified that he requested the records from the bank on "numerous occasions," but the bank refused to provide them directly to the Department or referred him to other branch offices. However, bank records are not the only way an employer can demonstrate the amount of payroll. This also can be established by business taxes or other records described in the BRR. Mr. Richardson denied knowing that business taxes are an option if bank records are unavailable.
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is RECOMMENDED that the Department of Financial Services, Division of Workers' Compensation, enter a final order finding that Respondent violated the workers' compensation laws by failing to secure and maintain required workers' compensation insurance for its employees, and imposing a penalty of $165,654.10. DONE AND ENTERED this 3rd day of May, 2019, in Tallahassee, Leon County, Florida. S D. R. ALEXANDER Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 Fax Filing (850) 921-6847 www.doah.state.fl.us Filed with the Clerk of the Division of Administrative Hearings this 3rd day of May, 2019. COPIES FURNISHED: Steven R. Hart, Qualified Representative Department of Financial Services 200 East Gaines Street Tallahassee, Florida 32399-4229 (eServed) Kyle Christopher, Esquire Department of Financial Services Hartman Building 2012 Capital Circle Southeast Tallahassee, Florida 32399 (eServed) Julie Jones, CP, FRP, Agency Clerk Division of Legal Services Department of Financial Services 200 East Gaines Street Tallahassee, Florida 32399-0390 (eServed) Adrian Shawn Middleton, Esquire Middleton & Middleton, P.A. 1469 Market Street Tallahassee, Florida 32312-1726 (eServed)
Findings Of Fact Based upon all of the evidence, the following findings of fact are determined: On April 19, 1990, petitioner, Don R. Blackburn, was a candidate on the engineering intern portion of the professional engineer examination given in Miami, Florida. The test was administered by the Department of Professional Regulation (DPR) on behalf of respondent, Board of Professional Engineers (Board). On July 25, 1990, the Board issued a written uniform grade notice advising petitioner that he had received a grade of 66 on the examination. A grade of 70 is necessary to pass this part of the examination. By letter dated August 15, 1990, petitioner requested a formal hearing to contest his score. In his letter, Blackburn generally contended that the examination was unfairly administered because certain books were allowed to be used by some but not all candidates, untrained proctors were given the authority to scan review materials and determine which could or could not be used by the candidates, and because of the chaos and confusion that occurred during the examination, he was unable to attain a score that he otherwise would have been able to achieve. Blackburn is an engineer for Lee County and is seeking to pass the engineering intern portion of the examination. A passing grade on that portion is a prerequisite to sitting on the second part of the professional engineer examination. He has taken the examination on a number of occasions and has gradually improved his score to just short of passing. Indeed, on the October 1989 examination, Blackburn scored a 69, or just one point less than the required 70. Prior to the April 1990 examination, the engineering intern portion of the professional engineer examination was an unrestricted open book examination. This meant candidates could use any and all reference and review materials during the examination. Beginning with the April 1990 examination, the Board imposed certain restrictions on the use of review materials. As early as October 9, 1989, the Board's executive director sent a memorandum to all candidates on the October 1989 examination, including Blackburn, concerning the new restrictions. The memorandum stated in part: Please be advised of certain restrictions listed in the Candidate Information Booklet which will not be implemented until the April 1990 examination. These restrictions are found in the "Examination Administration Information" section and are concerning the following two areas: * * * 2. Books or information containing sample questions or engineering problems may also be brought provided they are bound. Again, the new restrictions listed in the Candidate Information Booklet regarding the above two areas WILL NOT be implemented until the April 1990 examination. All candidates on the April 1990 examination were given a Candidate Information Booklet prepared in January 1990 by DPR's Bureau of Examination Services. On pages 13 and 14 of the booklet was found the following information: This is an open book examination. Candidates may use textbooks, handbooks, notes, and reference materials which are bound, copyrighted and printed. The term "bound" refers to material that is bound permanently, hard or paperback stitched or glued, or spiral, plastic or three-ringed bound. The printed material must remain contained (bound) in its cover during the entire examination. No writing tablets, unbound tablets or unbound "loose notes" will be allowed. No books with contents directed toward sample questions or solutions of engineering problems are permitted in the examination room. Examinees are not permitted to exchange reference materials or aids during the examination. (Emphasis in original) What the emphasized language meant is that "review" manuals, which contain problems and solutions, were prohibited from use during the examination while "reference" books were not. However, the booklet did not list the specific names of published materials that would be permitted or excluded. In order to ascertain which books he might use on the next examination, on March 27, 1990, Blackburn telephoned the Board in Tallahassee and spoke with a female employee named "B. J." who advised him that "review publications directed principally towards the solution of engineering problems" would be excluded. When asked if "Lindeburg's Sixth Edition" would be authorized, B. J. told Blackburn she wasn't sure and that it would be left up to the proctors in the room. She did say, however, that a review manual authored by Schaum could be used. The engineering intern examination in April 1990 was administered in two separate rooms at the Radisson Hotel in Miami, Florida. Blackburn was in a "very large" upstairs room with approximately thirty other candidates while a similar number took the examination in a downstairs room. The examination in the upstairs room began at 8:43 a.m. after various instructions were read to the candidates by the examination supervisor, Jeannie Smith, a veteran of twenty years in proctoring and supervising professional examinations. According to Smith, there was "considerable confusion" concerning which books could be used by the candidates, particularly since this was the first examination given with the new restrictions. She also acknowledged that there was "chaos" prior to the beginning of the examination and that this was, "extremely upsetting" to the examinees. However, before the examination began, Smith announced on a microphone the names of certain books which the Board had given her that were either prohibited or could be used by candidates. She further advised that if candidates had any questions they were to come to a bulletin board by the microphone where she had posted Xerox copies of the covers of various books. If a book could be used, it had the word "YES" printed on the cover while a "NO" was printed on those covers of books that could not be used. 1/ It is noted that only one cover sheet with a "YES" was posted, that being the Civil Engineering Reference Manual, Fourth Edition, Michael R. Lindeburg. However, at least three candidates who took the examination that morning, including petitioner, did not see the posted materials nor hear the invitation for candidates to come to the bulletin board. One book in issue that was specifically prohibited was Engineer In Training Review Manual, Sixth Edition, Michael R. Lindeburg, which contained 378 solved problems, and thus fell within the general prohibition of review manuals described on page 14 of the Candidate Information Booklet. However, those candidates who had the Seventh Edition of the same book were allowed to keep and use that manual even though it contained 422 solved problems, or some 44 more solved problems than was contained in the prohibited Sixth Edition. By allowing those students having the Seventh Edition to use the same even though it contained "review" materials, DPR violated the instructions contained in the Candidate Information Booklet and gave an advantage to those candidates not enjoyed by others, including petitioner. In addition, at least one other candidate in the upstairs group was allowed to use a prohibited review manual (Schaum's Outline Series, Theory and Problems of Electric Power Systems) but still that candidate did not attain a passing grade. Petitioner also contended that candidates taking the examination in the downstairs room were allowed to use language dictionaries during the morning part of the examination while those upstairs could not. 2/ Petitioner's contention is grounded upon hearsay evidence and accordingly it is found that no competent proof to support this claim was submitted. However, there was obviously some confusion over this matter because, after receiving complaints of this nature from two candidates, Smith telephoned the Board's offices in Tallahassee during the lunch break to ascertain whether such books could be used. Upon learning that they could not, she advised the upstairs group at the beginning of the afternoon session that dictionaries were not allowed. Blackburn also established that during the examination proctors went from desk to desk examining the materials that each candidate had in his possession. If a candidate had what the proctor perceived to be a book containing solutions to problems, the candidate was told to put the book on the floor. In the alternative, she candidates were told that if they tore the offending pages out of the book, they could continue using the remaining materials. Petitioner has complained that the proctors were not engineers and they were untrained in determining whether a book was acceptable or not. The Board has conceded that engineers do not proctor examinations but asserted that they are intelligent enough to determine whether books fall within the proscribed category. According to Blackburn's proctor at the examination, George Walton, a retired Coast Guard captain and engineering graduate of the Coast Guard Academy, he relied upon the list of approved and disapproved books supplied by the Board prior to the examination in determining whether materials would be excluded or not. Walton also stated that if he examined a book and found it contained solutions, he would disallow the same unless the offending pages were removed. A DPR expert in testing and measurements, Dr. Joseph A. Klock, examined the pass/fail rate for the examination taken by Blackburn and compared that rate to the October 1989 examination rate. Doctor Klock found no significant difference in the two rates and concluded that there was no statistically significant difference in performance of candidates over those time periods despite the confusion which occurred during the April 1990 examination. Blackburn did not present any evidence to show that if he had used the Seventh Edition of the Engineer In Training Review Manual, he would have been able to achieve more points on a particular problem and thus would have had a passing grade. Blackburn's principal complaint was that he had spent many hours preparing for the examination in question, that he was forced to guess which books to bring to the examination, and because of the confusion and chaos that took place at the beginning of the examination as well as his awareness that others were using a review manual with solved problems, it was impossible for him to give his best effort on the examination.
Recommendation Based on the foregoing findings of fact and conclusions of law, it is RECOMMENDED that petitioner's request to receive a passing grade on the April 1990 professional engineer examination be DENIED. However, petitioner should be entitled to retake the next examination at no charge. DONE and ENTERED this 28th day of November, 1990, in Tallahassee, Florida. DONALD R. ALEXANDER Hearing Officer Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, FL 32399-1550 (904) 488-9675 Filed with the Clerk of the Division of Administrative Hearings this 28th day of November, 1990.
Findings Of Fact 1. The Findings of Fact set forth in the Recommended Order are approved, adopted, and incorporated by reference as modified by Rulings on Respondent’s Exceptions noted above. 2. There is competent, substantial evidence to support the Findings of Fact.
Conclusions Based upon the foregoing findings of fact and conclusions of law, it is Ordered that Petitioner’s challenge to the licensure examination taken December 6, 2000, is Denied and his petition is Dismissed. This order takes effect upon filing with the Clerk of the Department of Health. Done and Ordered this ( , day of , 2002. BOARD OF PODIATRIC MEDICINE
Other Judicial Opinions A party who is adversely affected by this Final Order is entitled to judicial review pursuant to Section 120.68, Florida Statutes. Review proceedings are governed by the Florida Rules of Appellate Procedure. Such proceedings are commenced by filing one copy of a Notice of Appeal with the agency clerk of the Department of Health and a second copy, accompanied by filing fees prescribed by law, with the District Court of Appeal. That Notice of Appeal must be filed within thirty days of rendition of the order to be reviewed. CERTIFICATE OF SERVICE I HEREBY CERTIFY that a true and correct copy of the foregoing has been furnished by Certified Mail to Charles Pellegrini, Katz, Kutter, Alderman, Bryant & Yon, P.A., 106 E. College Ave., Suite 1200, Tallahassee, FL 32301, and Perry Verleni, 7624 S.W. 56th Avenue, Gainesville, FL 32608, and by interoffice mail to Cherry Shaw, Department of Health, 4052 Bald Cypress Way, Tallahassee, FL 32399-1783, Ella Jane P. Davis, Division of Administrative Hearings, 1230 Apalachee Parkway, Tallahassee, FL 32399-3060, and to Ann Cocheu, Office of the Attorney General, PL 01 The Capitol, Tallahassee, FL 32399-1050, this IS. day of "\ , 2002. LE qlee F.\Usens\ ADMIN\WILMA\ Ann \pod\000208d.wpd
The Issue The issue is whether Respondent properly denied Petitioner's application for a certificate of qualification, pursuant to Chapter 337, Florida Statutes, and Rule 14-22, Florida Administrative Code, for failure to timely file the application.
Findings Of Fact Petitioner is a family-owned construction firm located in Hattiesburg, Mississippi. It primarily acts as a contractor on pile driving projects. Petitioner is also involved in property management, oil, pre-stressed concrete, and general construction. Respondent selects its highway and bridge contractors from those who qualify under Section 337.14(1), Florida Statutes, and Rule 14-22.002(2), Florida Administrative Code. Petitioner has been a qualified bidder and construction contractor in Florida for Respondent continuously since the early 1940s. Contractors desiring to bid on state highway construction contracts in excess of $250,000 must apply annually to Respondent for a certificate of qualification. The application must be accompanied by the applicant's most recent annual financial statement, showing its financial condition no more than four months before Respondent receives the application. The application directs applicants to "mail" the completed forms to Respondent's Contracts Administration Office. Respondent sent Petitioner a Notice of Expiration of Qualification on or about February 3, 1999. The notice advised Petitioner of the following: Your qualification with this Department will expire on 04/30/99. Pursuant to Florida Statutes, your pre-qualification application must be 'filed' with the Department within four (4) months of the ending date of your financial statement. Filing is defined as receipt of the application by the Contracts Administration Office. Enclosed are two copies of the application form. Please return an original and one (1) copy of the application and all attachments. Please be advised all information must be filed in duplicate. In preparing your new application, please carefully follow the instructions on the application form and those enclosed with this notice. Additional reference material and a copy of Rule 14-22, F.A.C., are also enclosed for your convenience. Petitioner mailed its 1999 application for qualification, together with its most recent audit report dated December 31, 1998, to Respondent's Contracts Administration Office on April 27, 1999. Petitioner's fiscal year ends on December 31. An independent accounting firm begins preparing Petitioner's audit report shortly thereafter. The accounting firm's field work for Petitioner's 1998 audit was not complete until March 10, 1999, the date of the audit certification. This certification means there were no material changes in Petitioner's financial condition up to that date. There was no material change in Petitioner's financial condition over the holiday weekend from Thursday, December 31, 1998 to Monday, January 4, 1999. Changes in the amount of interest earned by Petitioner or charged against it during this brief interval would not make a difference in anyone's decision process. Any significant change up to March 10, 1999, would have been disclosed by the accounting firm in the December 31, 1998, audited financial statement. The same accounting firm has been auditing Petitioner's books for at least 15 years. Petitioner is in very sound financial condition. The Board of Governors of the United States Postal Service has set no more than three days as the standard for delivery of first class mail between Hattiesburg, Mississippi, and Tallahassee, Florida. This standard is not a guarantee. However, 90-94 percent of the time, the postal service delivers first class mail within two or three days throughout the country, as measured by an internal service measurement standard. The postal service also contracts with Price- Waterhouse to perform an independent external service measurement system called EXFC. Under that system, the postal service scored 100 percent on test mailings, delivered to Tallahassee from Mississippi within three days, over the past three years. The post office in Tallahassee, Florida, provides dedicated carrier service for the delivery of first class mail to state agencies five days a week. The postal service also delivers mail on Saturdays for the agencies that request weekend delivery. The postal service delivers Respondent's mail to its mailroom between 7:30 and 9:00 a.m. on weekdays. Respondent also receives mail on the weekends. Respondent's security guard pushes the weekend mail cart into the building on the weekends. Respondent's mailroom staff sorts incoming mail and places it in various bins for each mail station by 11:00 a.m. each week day. The mail stations send individuals down to the mailroom to pick up the mail from their assigned bin. If a station has not picked up its mail by 2:30 p.m., mailroom personnel deliver it to that station. Occasionally, first class mail is placed in the wrong bin and delivered to the wrong mail station. That mail is retrieved by the mailroom staff sometime after 2:30 p.m. and delivered to the correct office. At the end of the business day, no first class mail remains in the mailroom for delivery within the building. There is no evidence that first class mail intended for the Contracts Administration Office has ever been misdelivered. Bessie White, Administrative Assistant in the Contracts Administration Office, picks up the mail for her office from bin number 55. After she opens the mail, she stamps qualification applications and financial statements using an electric date and time clock. Ms. White saves envelope postal markings by cutting them out and attaching them to the applications. She then distributes the mail to various personnel in her office. The date and time clock that Ms. White uses to stamp incoming mail has to be reset manually when a calendar month ends in less than 31 days. April 30, 1999, was a Friday. Ms. White did not reset the date and time clock at the end of the business day. She did not work on the weekend of May 1-2, 1999. On Monday, May 3, 1999, Ms. White dated and time stamped Petitioner's application and financial statement before she reset the clock on the stamp machine. Consequently, the date and time stamp erroneously reflected that Petitioner's application and financial statement were received on Sunday, May 2, 1999, at 11:42 a.m. Ms. White distributed Petitioner's application and financial statement to the appropriate staff member in her office. Later that day, she reset the date and time clock to reflect the correct date of May 3, 1999. She did not go back to correct the erroneous stamps on mail, including Petitioner's documents, which she had already distributed. Ms. White discovered the error on Petitioner's application and financial statement seven months later, after Petitioner filed its protest in this matter. At that time, she made a handwritten notation on the documents, indicating their receipt on May 3, 1999. Petitioner mailed its 1999 application for qualification and its audited financial statements with a reasonable expectation that Respondent would receive them within three days, on or before April 30, 1999. Petitioner relied upon the postal service standard, and its own experience, in anticipating delivery of the documents within that time frame. Petitioner has an affiliate company located in Monticello, Florida. The Florida affiliate normally receives mail from Petitioner within three days. The record here contains no explanation for the delay by one business day in the receipt of Petitioner's application by Respondent's Contracts Administration Office. Petitioner mailed similar applications to Georgia, Arkansas, Missouri and Louisiana on April 27, 1999. All of these applications were approved. Respondent sent Petitioner a Notice of Intent to Deny Application for Qualification by certified mail on May 24, 1999. Respondent made this decision because the financial statements accompanying the application were dated more than four calendar months prior to the date the application was filed. Respondent counts calendar months in making its decision whether a qualification application is timely. In this case, Respondent took the position that the four-month time period ended on April 30, 1999, 120 days after December 31, 1999. 1./ If Petitioner's fiscal year had ended on June 30, 1999, Respondent would have required Petitioner to file its application within 123 days, on or before October 31, 1999. In determining whether a qualification application is timely, Respondent does not allow the five-day grace period for service by mail as set forth in Rule 28-106.403, Florida Administrative Code, and adopted by Respondent in Rule 14- 22.0011, Florida Administrative Code. Respondent approves over 400 qualification applications a year. It denies an average of 20 applications because they are late. When an application is denied as untimely, the applicant has an opportunity to furnish Respondent with an audited interim financial statement. An interim audit requires the same work and preparation as a regular annual audit. An interim audit would cost Petitioner between $25,000 and $30,000. Performing and filing an interim audit would also cause a three-month delay in the processing of Petitioner's application.
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is RECOMMENDED: That Respondent enter a final order approving Petitioner's application for a certificate of qualification. DONE AND ENTERED this 23rd day of December, 1999, in Tallahassee, Leon County, Florida. SUZANNE F. HOOD Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 SUNCOM 278-9675 Fax Filing (850) 921-6847 www.doah.state.fl.us Filed with the Clerk of the Division of Administrative Hearings this 23rd day of December, 1999.
