The Issue Whether Petitioner is entitled to additional credit for his solutions to Problems 124 and 222 of the Principles and Practice of Engineering portion of the engineering licensure examination administered on October 30, 1998, by the National Council of Examiners for Engineers and Surveyors.
Findings Of Fact Based upon the evidence adduced at hearing, and the record as a whole, the following findings of fact are made: On October 30, 1998, as part of his effort to obtain a Florida engineering license, Petitioner sat for the Principles and Practice of Engineering Examination (Examination). This is a national examination developed and administered by the National Council of Examiners for Engineers and Surveyors (NCEES). Petitioner chose to be tested in civil engineering. Petitioner received a raw score of 47 on the Examination. For the civil engineering specialization, a raw score of 47 converts to a score of 69. To pass the Examination, a converted score of 70 is needed. Petitioner formally requested the NCEES to rescore his solutions to Problems 124, 125, and 222 on the Examination. At the time he made this request, Petitioner was aware that rescoring could result in the candidate's score being lowered (although he believed that, in his case, the outcome would be a higher, not a lower, score). Petitioner was wrong. The rescoring he requested resulted in his receiving a raw score of 43 (or a converted score of 65, 5 points less than he needed to pass the Examination). After being notified of the outcome of the rescoring, Petitioner requested the Florida Board of Professional Engineers to grant him a "formal administrative hearing" on the matter. Petitioner's request was granted. At hearing, Petitioner advised that he was challenging only the grading of his solutions to Problems 124 and 222 of the Examination, and that he was not pursuing his challenge to the score he had received for his solution to Problem 125. Problems 124 and 222 were worth ten (raw) points each. Problem 124 contained four subparts (or requirements). Petitioner received two (raw) points for his solution to Problem 124. Rescoring did not result in any change to this score. Due to mathematical errors that he made, Petitioner did not solve any of the subparts of Problem 124 correctly. Accordingly, in accordance with the requirements and guidelines of the NCEES scoring plan for this problem, the highest (raw) score that he could have received for his solution to this problem was a two, which is the score he received. Problem 222 contained five subparts (or requirements). Petitioner originally received a (raw) score of six for his solution to Problem 222. Upon rescoring, his (raw) score was reduced to two. In attempting to solve Problem 222, Petitioner overestimated the lateral earth pressure due to his misunderstanding of the term "equivalent fluid pressure" used in the problem. In addition, in his solution to subpart (a), he did not properly specify the appropriate bar size and spacing. Giving Petitioner a (raw) score of two for his solution to Problem 222 was consistent with the requirements and guidelines of the NCEES scoring plan for this problem.
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is RECOMMENDED that a final order be entered rejecting Petitioner's challenge to the failing score he received from the NCEES on the Principles and Practice of Engineering portion of the October 30, 1998, engineering licensure examination. DONE AND ENTERED this 4th day of October, 1999, in Tallahassee, Leon County, Florida. STUART M. LERNER Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 SUNCOM 278-9675 Fax Filing (850) 921-6847 www.doah.state.fl.us Filed with the Clerk of the Division of Administrative Hearings this 4th day of October, 1999.
The Issue Whether Petitioner is eligible for licensure by endorsement as a professional engineer and/or waiver of Part I of the engineering licensing examination.
Findings Of Fact From 1969 through 1974, Petitioner attended the Lipetsk Branch of the Moscow Institute of Steel and Alloys. In 1974, Petitioner graduated from the Lipetsk Polytechnical Institute (Institute) in Russia, with a degree in industrial and civil engineering. The degree in civil engineering earned by Petitioner is equivalent to a bachelor of science degree in civil engineering in the United States. Upon graduation from the Institute, Petitioner commenced her professional employment as an engineer on August 30, 1974, at the LIPTSKGRAZHDANPROEKT Design Institute. In January 1976, Petitioner was employed as a professional engineer at the State Design Institute DNEPRPROEKKTSTALKONSTRUKSIYA (DNEPR), where she was continuously employed until leaving the Soviet Union in December 1993. There are no specific licensing or registration requirements in Russia with respect to engineers. Therefore, after earning a degree in industrial and civil engineering, Petitioner could work as an engineer in Russia without taking any professional examination. However, in 1979, Petitioner took an examination in Russia that allowed her to sign her own drawings and calculations. During Petitioner's tenure at the DNEPR, she achieved the status of Senior Engineer in 1986; was promoted to the position of Category II Engineer for Steel Structures in 1988; was promoted to the post of Category I Engineer for Steel Structures in 1990; and was elevated to the position of Leading Engineer in 1991. To achieve the status of Category II Engineer for Steel Structures and Category I Engineer for Steel Structures at DNEPR, Petitioner had to take an examination in 1988 and in 1990, respectively. The promotion to each of these positions was predicated upon Petitioner's passing these examinations and demonstrating expertise in the areas of economics, chemistry, mathematics, physics, building materials, corrosion prevention, resistance of materials, and construction mechanics. As a result of passing the examinations in 1988 and 1990, Petitioner was not only promoted, but also received salary increases. Petitioner believes that the two examinations she took in Russia in 1988 and in 1990, while working at the DNEPR were substantially equivalent to the Fundamentals Examination. However, no evidence was presented to support this claim. The Fundamentals Examination is one component of the engineering licensing examination, and is designed to assess whether an individual is qualified to practice in this state as an engineer intern. This examination is usually taken either in the applicant's last year in engineering school or shortly after graduation. With regard to format, the Fundamentals Examination is an eight-hour examination and consists of 120 multiple-choice questions. The Principles and Practice Examination is the second part of the engineering licensing examination and is taken after successful completion of the Fundamentals Examination. Oscar E. Olsen, a structural engineer and owner of O.E. Olsen and Associates, a structural engineering firm, is currently Petitioner's employer. Mr. Olsen, who is generally familiar with the Fundamentals Examination, testified that the list of subjects covered on the two examinations taken by Petitioner in 1988 and 1990, coincide with the subject matter on the Fundamentals Examination. Mr. Olsen further testified that it appeared to him that the two examinations taken by Petitioner were comparable to the Fundamental Examinations required in Florida. Notwithstanding his testimony that the exams taken by Petitioner are substantially equivalent to the Fundamentals Examination, Mr. Olsen admitted that he has never seen or reviewed the examinations taken by Petitioner while she was in Russia. It is impossible to render a reasonable opinion as to whether the two examinations taken by Petitioner in Russia are substantially equivalent to the Fundamentals Examination, where the only information provided with regard to the former is a list of subject areas covered. Such a list gives no indication of the depth and specific content of the subject matter on the examinations; the difficulty of the examinations; the passing scores; the number and format of the questions; and the length of the examinations.
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is recommended that Respondent, the Board of Professional Engineers, enter a Final Order denying Petitioner's request for waiver of Part I, the Fundamentals Examination, and for licensure by endorsement. DONE AND ENTERED this 9th day of December, 1997, in Tallahassee, Leon County, Florida. CAROLYN S. HOLIFIELD Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (904) 488-9675 SUNCOM 278-9675 Fax Filing (904) 921-6847 Filed with the Clerk of the Division of Administrative Hearings this 9th day of December, 1997. COPIES FURNISHED: Murray Silverstein, Esquire Powell, Carney, Hayes, and Silverstein One Plaza, Suite 1210 St. Petersburg, Florida 33731-1689 Edwin A. Bayo Assistant Attorney General Office of the Attorney General The Capitol, Plaza 01 Tallahassee, Florida 32399-1050 Angel Gonzalez Executive Director Board of Professional Engineers Northwood Centre 1940 North Monroe Street Tallahassee, Florida 32399-0755 Lynda L. Goodgame, General Counsel Department of Business and Professional Regulation 1940 North Monroe Street Tallahassee, Florida 32399-0792
The Issue The issue for determination is whether Respondent committed the offenses set forth in the Second Amended Administrative Complaint and, if so, what action should be taken.
Findings Of Fact At all times material hereto, Mr. Danforth was licensed by the Board as a professional engineer, having been issued license number PE 44653. On March 5, 1996, Mr. Danforth was issued a Special Inspector license, license number 1103, by the Department of Community Affairs (DCA). Mr. Danforth submitted an application for the Special Inspector's license (Application) to DCA on January 29, 1996, based upon his experience and hours of structural design course work. In his Application, Mr. Danforth represented that, during 1995 through 1996, he had inspected the structural elements of four threshold type buildings. From October 1993 through April 1998, Mr. Danforth was employed by Fraser Engineering & Testing, Inc. (FET). The owner of FET, Alexander Fraser, who, at the times material hereto, was a licensed engineer, testified at hearing. Mr. Fraser's testimony is uncontroverted and found to be credible. FET performed threshold inspections as part of its business, but such inspections comprised a minority part of its business. FET did not engage in any structural design. Mr. Danforth was the most senior geotechnical engineer for FET in charge of foundation investigations. Mr. Fraser was the Special Inspector in charge of and responsible for the projects referred to in Mr. Danforth's Application. According to Mr. Fraser, Mr. Danforth supervised all threshold projects for FET and periodically visited project sites, including the four threshold projects in Mr. Danforth's Application. As to structural engineering special inspections by FET, Mr. Fraser testified that Mr. Danforth supervised FET's inspectors, who performed the day-to-day inspections, and would perform periodic site inspections to make sure that the day-to- day inspections were being properly made by the inspectors. Mr. Fraser further testified that to make certain that the inspectors were performing in accordance with "proper engineering," Mr. Danforth had to be "intimately involved in the inspections" by making on-site visits, discussing the projects with the inspectors when they returned at night, and reviewing the daily reports generated on the sites. If the inspectors who performed the day-to-day inspections had questions or problems, they went to Mr. Danforth. At the time for the final reports, Mr. Danforth would sign the final reports, followed by Mr. Fraser who would sign them as the Special Inspector. Mr. Fraser had complete confidence in Mr. Danforth. The former operations manager of FET, David Alker, who was employed with FET during the time period of the four threshold projects, testified that all structural inspection issues, regarding the projects, were referred to Mr. Danforth. Mr. Alker's testimony is found to be credible. The Board presented the testimony of an expert witness, Jose R. Danon, Ph.D., P.E., who was a structural engineer. Among other things, Mr. Danon reviewed Mr. Danforth's Application to DCA for the Special Inspector's license, documents regarding Mr. Danforth's engineering experience, and Florida Administrative Code Rule 9B-3.043, which was the rule, at the time of Mr. Danforth's Application, setting forth the minimum qualifications for the Special Inspector's license. Mr. Danon testified that Mr. Danforth's Application material did not provide sufficient information for him (Mr. Danon) to make a determination as to whether Mr. Danforth was qualified for a Special Inspector's license; and that the description of Mr. Danforth's engineering experience was insufficient to make a determination as to whether Mr. Danforth had sufficient engineering experience to qualify for a Special Inspector's license. Accordingly, Mr. Danon opined that Mr. Danforth failed to meet the minimum qualifications for a Special Inspector's license. Mr. Danon's opinion is not found to be credible. He is not a Special Inspector and has never applied for or been issued a Special Inspector's license. Mr. Danon essentially testified that, because of the lack of sufficient information, he was unable to make a determination that Mr. Danforth was qualified to be licensed as a Special Inspector. No testimony was presented by Mr. Danon applying the facts, as he knew them from the documents provided to him by the Board, to the minimum qualifications set forth in Florida Administrative Code Rule 9B- 3.043. Mr. Danon had no knowledge as to how DCA interpreted the said Rule. Moreover, Mr. Danon had no knowledge as to what criteria DCA used to issue Mr. Danforth the Special Inspector's license. Mr. Danon did not opine in any way, form or fashion that Mr. Danforth submitted false, fraudulent, or misleading information or negligently submitted information regarding his Application and/or engineering experience. Mr. Danon testified that he had no opinion, that he did not know, as to whether the information was false, fraudulent, or misleading. A finding is made that the evidence fails to show that Mr. Danforth submitted false, fraudulent, or misleading information or negligently submitted information for his Special Inspector's license. Mr. Danon heard the testimony of all the witnesses, including Mr. Danforth. Having heard all of the testimony, Mr. Danon was not called or re-called to testify as to whether his opinion had changed regarding whether Mr. Danforth met the minimum qualifications for a Special Inspector's license in 1996. The evidence does not show that DCA failed to properly apply the minimum criteria of Florida Administrative Code Rule 9B-3.043. Mr. Danforth was also a former employee of Professional Engineering and Inspection Company, Inc. (PEICO). He was one of PEICO's professional engineers and Special Inspectors and had become one of its officers. On January 27, 2003, Mr. Danforth signed and sealed a Threshold Inspection Completion Statement (Completion Statement), as Special Inspector, for a PEICO project, the Jefferson at Camino Real, located in Boca Raton, Florida. The Completion Statement was submitted to the City of Boca Raton Building Department. Mr. Danforth certified in the Completion Statement that "To the best of our knowledge and belief, the construction of all structural load bearing components described in the threshold plan complies with the permitted documents, and the shoring and reshoring conformed with the shoring and reshoring plans submitted to the enforcing agency." Also, on January 27, 2003, Mr. Danforth signed and sealed a Special Inspector Certificate (Certificate), as Special Inspector for the PEICO project. The Certificate was also submitted to the City of Boca Raton Building Department. The certificate provided Mr. Danforth, with certifying that he agreed “to be present on the project at all times when construction activities for which I am Special Inspector, to check the work for compliance with approved drawings and applicable codes and ordinances of the City of Boca Raton, and to submit both periodic reports and the Certificate of Completion . . ." Mr. Danforth was not the original Special Inspector on the PEICO project. Gary H. Elzweig, P. E., was the original Special Inspector for the PEICO project, but he had been fired and was in litigation with PEICO. Mr. Danforth was not on the project as a Special Inspector, so he had not been making periodic on-site inspections, discussing the project with the day-to-day, on-site inspectors, and reviewing reports. Mr. Danforth testified that, prior to signing the Completion Statement and the Certificate, he conferred with the Board and the City of Boca Raton Building Department to determine what were the necessary requirements to change the qualifying Special Inspector on the project and certify the project. He testified that the Board advised him that the requirements of the local jurisdiction, i.e., the City of Boca Raton, controlled. Mr. Danforth further testified that the City of Boca Raton Building Department advised him that the same documents submitted by the original Special Inspector at the beginning of the project must be re-submitted by the new Special Inspector and that the Completion Statement must be submitted by the new Special Inspector. Mr. Danforth's testimony is found to be credible. Furthermore, prior to signing the Completion Statement, Mr. Danforth acted as he thought was "prudent and within the standard duty of care to be able to sign-off" on the project. He reviewed the inspection reports; interviewed all the day-to-day, on-site inspectors to make sure that no remaining issues existed regarding the threshold inspection; and visited the project himself, making an on-site inspection. Mr. Danforth was satisfied that he could execute the Completion Statement. On February 3, 2003, Mr. Danforth submitted to the City of Boca Raton Building Department a revised Completion Statement. He certified that Mr. Elzweig was no longer employed by PEICO and no longer the Special Inspector; that he (Mr. Danforth) was replacing Mr. Elzweig as the Special Inspector; and that "To the best of our knowledge and belief, the construction of all structural load bearing components described in the threshold plan complies with the permitted documents, and the shoring and reshoring conformed with the shoring and reshoring plans submitted to the enforcing agency." Mr. Danon opined that "any qualified Special Inspector (SI) working for the company PEICO could have signed the Certificate of Completion (COC) for their project as long as the SI had reviewed and concurred with the documentation Mr. Elzweig had approved during his working relationship with PEICO." Furthermore, Mr. Danon opined that Mr. Danforth committed no wrongdoing in signing the COC. Mr. Elzweig testified that Mr. Danforth committed no wrongdoing in signing the COC. The signing of the Completion Statement, or the COC, and the revised Completion Statement by Mr. Danforth is found to be in compliance with the requirements of the local jurisdiction.
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is RECOMMENDED that the Department of Business and Professional Regulation, Board of Professional Engineers enter a final order dismissing the Second Amended Administrative Complaint. DONE AND ENTERED this 11th day of April 2005, in Tallahassee, Leon County, Florida. S ERROL H. POWELL Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 SUNCOM 278-9675 Fax Filing (850) 921-6847 www.doah.state.fl.us Filed with the Clerk of the Division of Administrative Hearings this 11th day of April, 2005.
The Issue The issue in this case is whether Florida Administrative Code Rule 61G15-21.004(2) is an invalid exercise of delegated legislative authority.
Findings Of Fact The Parties Petitioner is an applicant for licensure as a professional engineer ("P.E.")2 in Florida, and currently works in the discipline of environmental engineering in Florida. His practice focuses primarily on water-related areas within that discipline. Petitioner is not currently licensed as a P.E. Respondent is a board within the Department of Business and Professional Regulation ("Department"). It is the state agency created pursuant to section 471.007, Florida Statutes, and charged with licensing professional engineers in Florida. Respondent is vested with the authority to adopt rules to implement chapter 471, regarding the regulation of the practice of engineering in Florida, as defined in section 471.005(7). Respondent adopted the Challenged Rule at issue in this proceeding. Statutory and Rule Background The engineering profession in Florida is regulated pursuant to chapter 471. A person may become licensed as a P.E. in Florida by applying for licensure, fulfilling specified educational and experience requirements, and 2 For purposes of this Final Order, the terms "professional engineer" or "P.E." will be used to refer to persons who are licensed engineers under chapter 471, Florida Statutes. either being endorsed for licensure as provided in sections 471.015(3) and (5), or passing the required licensure examinations. § 471.015, Fla. Stat. Pursuant to section 471.015, Respondent has adopted Florida Administrative Code Chapter 61G15-20, which codifies, in rule, the requirements for licensure as a P.E. in Florida. An applicant for licensure must be a graduate of a Board-approved engineering program; have the requisite number of years of engineering experience; and have passed the specified licensure exams. Fla. Admin. Code R. 61G15-20.0010. Section 455.217(1)(d) authorizes Respondent to adopt, by rule, the use of a national professional licensing examination that the Department has certified as meeting requirements of national examinations and generally accepted testing standards. To implement section 455.217(1), Respondent has adopted rule 61G15- 21.001, titled "Examination Designated; General Requirements." This rule requires that, unless an applicant qualifies for licensure by endorsement, he or she must pass the National Council of Examiners for Engineering and Surveyors ("NCEES") licensure exam. Part I of the NCEES exam is the Fundamentals of Engineering ("FE") exam, and part II of the NCEES exam is the Principles and Practice Exam ("PP") exam. Respondent has entered into a contract with NCEES to provide the FE and PP exams in Florida. A person must pass both the FE and PP exams to be licensed as a P.E. in Florida. § 471.015(1), Fla. Stat. The Challenged Rule states: "[t]he passing grade for Principles and Practice Exam is determined by the National Council of Examiners for Engineering and Surveying, where psychometric statistical methods are used to determine the level of performance that corresponds with minimal competence in the discipline." Fla. Admin. Code R. 61G15-21.004(2). The Challenged Rule is a subsection of rule 61G15-21.004, which is titled "Passing Grade." The Challenged Rule specifically and exclusively addresses the method for determining the passing grade on the PP exam. Sections 455.217(1)(d) and 471.013 are cited as the rulemaking authority for the Challenged Rule, and sections 455.217(1)(d) and 471.015(1) are cited as the law implemented by the Challenged Rule. The term "engineering," as used in section 471.005(7), includes the term "professional engineering," and defines the types of services and creative work that constitutes "engineering." An "engineer," as defined in section 471.005(5), includes the terms "professional engineer" and "licensed engineer," and means a person who is licensed to engage in the practice of engineering under chapter 471. By contrast, an "engineer intern," as defined in section 471.005(6), means a person who has graduated from a Board-approved engineering curriculum and has passed the FE exam. By definition, these are distinct terms. The term "engineer" is used to describe a person licensed as a P.E. under chapter 471, while the term "engineer intern" is used to described a person who may engage in the kinds of activities described within the term "engineering," as defined in section 471.005(7), but who is not licensed as a P.E. in Florida, and, therefore, is not authorized to hold himself or herself out as a licensed engineer in Florida. The Rule Challenge Petition The Rule Challenge Petition alleges four grounds under section 120.52(8) for invalidating the Challenged Rule. Alleged Invalidity of Challenged Rule under Section 120.52(8)(c) In paragraph 15 of the Rule Challenge Petition, Petitioner asserts that section 455.217(1)(c)—which, at the time the Rule Challenge Petition was filed, was cited as the specific authority for, and law implemented by, the Challenged Rule—did not authorize the Challenged Rule, so that the Challenged Rule enlarged, modified, or contravened the specific provisions of law implemented, pursuant to section 120.52(8)(c). As a result of Petitioner having filed the Rule Challenge Petition, Respondent discovered that it had not updated its citation of the specific authority for, and law implemented by, the Challenged Rule, when section 455.217(1) was amended and renumbered in 1997, so that section 455.217(1)(c) no longer was the correct citation to the law implemented by the Challenged Rule. Respondent requested the Department of State, Administrative Code and Register Section ("DOS"), to make a technical, non-substantive change to the Challenged Rule. As authorized by Florida Administrative Code Rule 1-1.010(10), DOS updated the statutory citation to section 471.217(1)(d), which is the correct citation to the law implemented by the Challenged Rule. This technical change nullifies the alleged invalidity ground set forth in paragraph 15 of the Rule Challenge Petition, and Petitioner concedes this. In paragraph 16 of the Rule Challenge Petition, Petitioner also alleges that the Challenged Rule enlarges, modifies, or contravenes section 455.217(1)(a), because the PP examination does not adequately and reliably measure an applicant's ability to practice the profession regulated by the Department. However, as discussed below, section 455.217(1)(a) is not cited as a specific provision of law implemented by the Challenged Rule, so cannot form the basis of a challenge to the Rule under section 120.52(8)(c). Alleged Invalidity of Challenged Rule under Section 120.52(8)(d) In paragraph 17 of the Rule Challenge Petition, Petitioner alleges that the Challenged Rule is invalid under section 120.52(8)(d) because it is vague, fails to establish adequate standards for agency decisions, or vests unbridled discretion in the agency. In support of this alleged invalidity ground, Petitioner asserts that the Challenged Rule is vague because "the level of performance on the PP exam is stated to correspond with minimal competency, yet there are no rules which provide definitive guidance to NCEES on what constitutes the general areas of competency in regards to engineering practice."3 Petitioner also alleges, in paragraph 19 of the Rule Challenge Petition, that the Challenged Rule is invalid under section 120.52(8)(d) because it equates passage of the PP exam with a 30-year practice experience requirement for licensure by endorsement set forth in section 471.015(5)(b). To this point, Petitioner states: "I can't think of anything more arbitrary than the principles and practice exam equating to near[-]retirement level experience."4 Alleged Invalidity of Challenged Rule under Section 120.52(8)(e) In paragraph 18 of the Rule Challenge Petition, Petitioner alleges that the Challenged Rule is invalid pursuant to section 120.52(8)(e) because it is arbitrary, for several reasons. Paraphrased, these reasons are: passage of the PP exam does not accurately reflect, or equate to, minimal competence in the discipline; the PP Exam does not accurately evaluate an individual's engineering ability level, but instead evaluates an individual's exam performance compared to average group exam performance; the PP exam does not reliably distinguish between minimal competence and incompetence to practice engineering, as evidenced by the fact that engineers who fail the PP exam still competently perform, and, thus, keep, their engineering jobs; passing the PP exam, by itself, does not certify an individual to competently perform any engineering service or creative work as defined in 3 Section 455.217(1)(b) requires, for each exam developed by the Department or a contracted vendor, that the general areas of competency covered by the exam be specified by rule. The last sentence of that subsection states that the requirements of subsection (b) do not apply to national exams, such as the NCEES PP exam, which are approved and administered pursuant to section 455.217(1)(d). Thus, the law implemented by the Challenged Rule does not require areas of competency to be specified in the Challenged Rule. 4 Because paragraph 19 of the Rule Challenge Petition alleges that the rule is arbitrary, the undersigned considers this paragraph to constitute a challenge to the Challenged Rule under section 120.52(8)(e), rather than under section 120.52(8)(d), as cited in the Rule Challenge Petition, and has addressed this ground in the Conclusions of Law section dealing with that alleged invalidity ground. section 471.005(7); the PP exam does not reliably determine if an examinee is minimally competent, due to an incorrect reference point; and the PP exam does not reliably distinguish between individuals whose practice of engineering would protect the public health and safety and those whose practice of engineering would constitute a danger to public health and safety. Alleged Invalidity of Challenged Rule under Section 120.52(8)(f) In paragraph 20 of the Rule Challenge Petition, Petitioner alleges that the Challenged Rule is invalid pursuant to section 120.52(8)(f) because it imposes regulatory costs on the regulated person, county, or city, which could be reduced by the adoption of less costly alternatives that substantially accomplish the statutory objectives. To this point, Petitioner proposes a state- conducted investigation of an applicant as a substitute for the PP exam, and contends that "there's a premium associated with a national exam. It's also clear that the regulatory cost imposed on me and all future examinees could be substantially reduced if the Board conducted their exam as an investigation and did not incorporate the NCEES exam." However, as more fully discussed below, this challenge ground is time- barred by section 120.541(1)(g), and, therefore, is not a legally-cognizable basis for invalidating the Challenged Rule in this proceeding. The Parties' Stipulated Facts Petitioner is an Engineer Extern, Texas EIT 56990. Pursuant to section 471.015(1), the Florida Engineers Management Corporation "shall issue a license to any applicant who the Board certifies is qualified to practice engineering and who has passed the [FE] exam and the [PP] exam." Both the FE exam and the PP exam are created by NCEES, pursuant to section 455.217(1)(d), which states, in pertinent part: "a board . . . may approve by rule the use of any national examination which the [Department of Business and Professional Regulation] has certified as meeting the requirements of national examinations and generally accepted testing standards pursuant to department rules." Pursuant to section 471.015(1), Petitioner passed the FE exam on September 3, 2016. Passing the FE is a prerequisite to take the PP exam. Petitioner registered to take the NCEES computer-based PP exam for environmental engineering on April 22, 2020. He paid the $350 exam fee and additional monies for test preparation material, and spent at least 100 hours preparing for the exam. He was notified, on April 6, 2020, by Pearson Vue, the test center company, that the exam was cancelled due to the Covid-19 pandemic. On April 7, 2020, Petitioner registered to take the exam on July 15, 2020, which was the earliest available date for taking the exam in his local area. On April 9, 2020, Petitioner canceled his registration for the July 15, 2020, exam and decided to apply for licensure as a P.E. without passing the PP exam. Pursuant to section 471.015(2)(a)1., on May 18, 2020, Petitioner filed his application for licensure with the Board. The application provides information stating that Petitioner meets the requirements of section 471.013(1)(a)1., and has at least four years of active engineering experience of a character indicating competence to be in responsible charge. In compliance with the education requirements of section 471.013(1)(a)1., Petitioner earned a Bachelor of Science degree in Chemical Engineering from the University of Florida, which is accredited by ABET. Petitioner provided an official transcript to the Board for verification purposes. In compliance with rule 61G15-20.002, Petitioner states in his application that he has at least four years of acceptable engineering practice, consisting of one year of experience equivalent through his Master of Engineering degree from the University of Texas at Austin, an ABET- accredited institution; and over three years of professional engineering work verified by licensed engineers. According to rule 61G15-20.002, an applicant must list three current personal references who are professional engineers. Thirteen licensed engineers submitted documentation to the Board regarding Petitioner's type of qualifying experience, level of engineering competency, and professional integrity. All references circled "yes" to the question "would you employ the applicant in a position of trust?" Two additional references in the engineering industry also provided their opinion on Petitioner's integrity and competence. On June 1, 2020, Petitioner received notification from the Board that his Florida 0901 1031-P.E. Endorsement application was incomplete. Petitioner addressed all items listed in a timely fashion to participate in the June 10, 2020, Board meeting. The Board stated during an informal hearing on June 10, 2020, that Petitioner would not be granted P.E. licensure due to not having passed the PP Exam. Petitioner is substantially affected by the Challenged Rule because it disqualifies him from becoming a licensed engineer in Florida without passing the PP exam. Findings of Fact Based on Evidence Presented at the Final Hearing A. Findings Regarding the Evidence Presented in Petitioner's Case Testimony of Witnesses Hoot and Grace David Hoot and Nigel Grace, both of whom are licensed professional engineers, testified regarding Petitioner's abilities and skills as a practicing engineer. Neither Hoot nor Grace was qualified as an expert in the field of psychometrics or related topics. Therefore, any opinions regarding these subject matters to which they testified at the hearing have been treated as personal opinions, rather than expert opinions. Hoot characterized Petitioner as a good, diligent young engineer, and described Petitioner's role in various projects on which they worked together. Based on his work with Petitioner, it was Hoot's personal opinion that Petitioner possesses the integrity and competence to work as a licensed engineer who would serve the public health and safety. Hoot testified that when he took the PP exam approximately 38 years ago, it was a free response exam. He stated that he was "not exactly sure" that a multiple-choice exam captures an examinee's ability to apply reason and judgment, but he acknowledged that he does not know how the PP exam currently is developed. He offered his personal opinion that it was possible for a competent engineer to fail the PP exam. Hoot also offered his personal view that engineering licensure constitutes a standard of care; however, he did not think anything provides a guarantee of competence. It was his personal view that experience tended to make one more competent. He also offered his personal view that as an engineer gains more experience and becomes more specialized, it is understandable that he or she would not score as well as an engineer who is gearing up toward the four-year experience goal which enables them to apply for licensure as a P.E. As Hoot put it, "life happens. You have children involved. You move jobs. I think you . . . maybe have less time to study. . . you get farther away from the study habits of . . . learning to be able to take tests." Grace, who is employed as a P.E. with Brown and Caldwell ("B&C"), a large U.S. engineering firm, testified regarding Petitioner's work experience while he was employed by B&C. Petitioner's experience included working on drinking water projects, utilities, upgrading process equipment, site evaluations, bench scale testing, and other projects. Based on Grace's work with Petitioner, it was his personal opinion that Petitioner possesses the integrity and competence to work as a licensed engineer who would serve the public health and safety. Grace took the PP exam approximately 28 years ago, and at the time, a major portion of the exam consisted of long-form written exam questions that provided the opportunity for examinees to provide free response answers and earn partial credit for partially-correct answers. He testified that the exam also had a multiple-choice component. Grace testified that, "based on instinct," he knows well-designed multiple-choice questions can provide the same insight into an examinee's decision-making judgment as long-form questions. Grace's personal view is that passing the PP exam does not guarantee competence, and it is possible for an engineer to be competent in some engineering disciplines but not others. Grace also agreed that it was logical that an examinee with approximately four years' experience would perform better on the PP exam because he or she would have better-honed test-taking skills and be fresher in some areas tested on the exam. By contrast, engineers who have practiced longer have more experience, but often have become specialized and further removed from the test-taking environment. Thus, passing a broad-spectrum exam becomes a bigger hurdle for engineers who have practiced longer. Petitioner's Testimony Petitioner testified on his own behalf at the final hearing. As discussed above, Petitioner holds a bachelor of science degree in chemical engineering from an ABET-accredited institution, and holds a master of engineering degree from an ABET-accredited institution. Although Petitioner, through his training as an engineer, is skilled at mathematics and statistics, he is not trained, and does not have any substantial experience, in the field of psychometrics.5 5 As discussed in greater detail below, psychometrics is a specialized field of study that concerns the theory and technique of objective psychological measurement of skills, knowledge, abilities, and educational achievement. Petitioner acknowledged that he does not have a degree in psychometrics and that he is not trained in, or knowledgeable about, preparing and administering high-stakes professional examinations. At the time he filed the Rule Challenge Petition, Petitioner had no personal knowledge of the work done by psychometricians; did not know what a "cut score" was; and was not familiar with the Modified Angoff Method, item response theory, the specifics of converting raw scores to scaled scores, or other any psychometric tools and methods employed to prepare and score the PP exam. Petitioner acquired some rudimentary knowledge in a few of these areas in preparing for the final hearing in this proceeding.6 Section 90.701(2), Florida Statutes, prohibits a lay witness from testifying as to an opinion regarding a matter involving specialized knowledge.7 Accordingly, it is determined that Petitioner is not competent to provide an expert opinion regarding psychometrics and related areas, such as item response theory. He was not qualified, tendered, or accepted at the final hearing as an expert in psychometrics or any related areas. Because Petitioner was not qualified to testify as an expert witness at the final hearing, Petitioner's testimony regarding psychometrics, item response theory, scaled scores, the Modified Angoff Method, high-stakes professional testing, and all other specialized subject matters, consisted of opinion testimony by a lay witness. The only instances in which a lay witness 6 Petitioner does not have any special knowledge, formal training, education, or experience in the specialized field of psychometrics. His knowledge about these areas was acquired by reading and study in preparation for the final hearing. Because Petitioner lacks special knowledge, experience, training, and education in psychometrics, he is not qualified to testify as an expert in psychometrics or related topics, such as item response theory. See Chavez v. State, 12 So. 3d 199, 205 (Fla 2009)(in determining whether a witness is qualified to render an opinion as an expert in a specialized field, the court must determine whether the witness is adequately qualified to render an opinion based on special knowledge, experience, training, or education). 7 Chesser v. State, 30 So. 3d 625 (Fla. 1st DCA 2010)(it is error for a court to accept opinion testimony of a lay witness in a specialized subject matter area). may provide opinion testimony are when the lay witness's opinion is based on firsthand knowledge through personal perception.8 As the undersigned explained during the final hearing, rather than excluding Petitioner's lay opinion testimony, she would consider these pertinent evidentiary principles in determining the weight to be afforded Petitioner's lay opinion testimony in this proceeding. Applying these evidentiary principles, it is determined that Petitioner's testimony regarding psychometrics and related topics, such as item response theory, classical response theory, and high-stakes test reliability and validity, concerned specialized subject matters not within the realm of common knowledge or based on Petitioner's personal perception. Rather, such specialized subject matters required expert witness testimony, pursuant to section 90.702, and as addressed above, Petitioner was not shown to be an expert in any of these specialized subject matters. Because Petitioner's testimony constituted the type of opinion testimony that is not permissible by a lay witness, pursuant to section 90.701, such testimony is not afforded weight in this proceeding. Petitioner testified that "minimal competence," which is the standard measured on the PP Exam, equates to "competence" as defined in the dictionary—that is, the "sufficient ability for a specific need or requirement." However, this position ignores that, for purposes of the Challenged Rule, "minimal competence" is a term of art specifically used, in the psychometric measurement context, to describe the level of competence that corresponds to a passing score on the PP exam. As more fully discussed below, the PP exam is developed and scored using psychometric tools and methods. 8 Nat'l Commc'ns. Indus., Inc. v. Tarlini, 367 So. 2d 670, 671 (Fla. 1st DCA 1979)(lay witness testimony regarding a specialized subject matter was not admissible into evidence because the testimony was not regarding a subject matter about which the witness could testify based on common knowledge or his personal perception.) Petitioner contends that engineering experience is, by itself, a reliable measure of competence, so it is unnecessary to also pass the PP exam. Petitioner testified "the application process and [PP] exam have two entirely different methods to identify the same result: whether an engineer in training is competent enough to become a licensed engineer It is not logical for two checks of competence to come up with different results. There should be consistency." In support of this position, Petitioner relies on section 471.015(2)(a), which requires "at least 4 years of active engineering experience of a character indicating competence to be in 'responsible charge' of engineering." § 471.015(2)(a), Fla. Stat. "Responsible charge" is defined in rule 61G15-18.011(1) as the degree of control an engineer is required to maintain over engineering decisions made personally or by others over which the engineer exercises supervisory direction and control authority. An engineer in responsible charge is the "engineer of record," as defined in rule 61G15-30.002(1). Rule 61G15-30.002(1) defines "engineer of record" as a Florida professional engineer who is in responsible charge. Thus, an engineer who is qualified, for purposes of being in responsible charge pursuant to section 471.015(2)(a) must, in addition to having the minimum statutory experience, be a licensed P.E. This means that he or she necessarily must have passed the PP examination. These statutory and rule provisions collectively reinforce the point that for an engineer to demonstrate competence for purposes of holding himself or herself out as an "engineer," as defined in section 471.005(7), he or she must satisfy all three requirements of section 471.015(2)(a)—i.e., education, experience, and passing the licensing exam.9 9 The requirement to meet these three requirements, including the PP exam, is codified in section 471.051(2)(a). Eliminating the exam requirement and relying strictly on education and/or experience for licensure would require the Florida Legislature to amend this statute Petitioner echoed the testimony of Hoot and Grace that licensure is not a guarantee of competence, and that passing the PP exam does not guarantee minimal competence. To this point, he testified that he does not believe that the PP exam adequately and reliably measures an applicant's ability to practice engineering, and that experience is a better indicator of competence than passing the exam. By way of example, Petitioner described his own experience10—which he characterized as "directly matching" the activities in which a licensed engineer engages—and compared that experience to measuring competence by an exam, which Petitioner characterized as "attempt[ing] to indirectly measure my ability as an engineer." Based on his personal experience, Petitioner contends that experience better demonstrates competence to be licensed as a P.E.; that passing the PP exam does not indicate minimal competence to practice engineering; and that failing the PP exam does not mean that the examinee is not minimally competent. He further testified that examinees who fail the PP exam likely are minimally competent, since the engineering jobs they hold when applying for licensure likely would require that they be minimally competent in order to have been hired.11 to eliminate the exam requirement. The undersigned is not authorized by statute or the Florida Constitution to eliminate the PP exam requirement for licensure under chapter 471. 10 Petitioner's experience, set forth in his P.E. licensure application, was verified by his supervising engineers. 11 Petitioner appears to conflate being determined not "minimally competent" for purposes of passing the PP exam, with "incompetence," which is defined in Florida Administrative Code Rule 61G15-19.001(5) as the "physical or mental incapacity or inability of a professional engineer to perform the duties normally required of the professional engineer." Part of this confusion may be due, in part, to Respondent's response to one of Petitioner's interrogatories asking for a definition of "minimal competence." Rather than directly answering the interrogatory, Respondent referred Petitioner to the definition of "incompetence" for purposes of imposing discipline under Respondent's disciplinary rules—thus causing Petitioner to understandably assume that failing to demonstrate minimal competence through passing the PP exam equates to "incompetence," as defined in rule 61G15-19.001(5). However, the fact that Petitioner has not demonstrated "minimal competence" on the PP exam does not mean that he is incompetent; it simply means that he has not yet passed the PP exam for licensure as a P.E. in Florida. To this point, if failing to demonstrate "minimal competence" by passing the PP exam equated to being incompetence, every person who performs engineering work in To further illustrate this point, Petitioner noted that the data regarding passage rate of the PP exam shows that examinees having zero years of experience are almost twice as likely to pass the PP exam as examinees having 11 or more years of experience. However, as Hood and Grace explained, and as further discussed below, engineers having more than four to five years of experience begin to specialize in narrower fields and "life happens," in that personal and professional circumstances render it more difficult to prepare for and take a high-stakes test. Petitioner also disputed the accuracy of the PP exam preparation and scoring process. In particular, he took issue with the "model law engineer" standard to which the exam is designed. As discussed more extensively below, this standard equates to the competence level of an engineer having four years of engineering experience and who is capable of practicing engineering in a manner that protects the public health and safety. In particular, Petitioner contends that designing the PP exam to the "model law engineer" standard is unfair to anyone taking the exam that does not have exactly four of years of engineering experience. Notably, however, section 471.015(1), which is the statute implemented by the Challenged Rule, establishes four years as the engineering experience required for licensure as a P.E. Thus, the "model law engineer" standard is rationally related to the statutory minimum experience level for purposes of demonstrating minimum competency to be licensed. Petitioner also contends that the subject matters tested on the PP exam are unfairly broad, so that engineers who specialize in a particular area within an engineering discipline—such as specializing in water-related areas in environmental engineering—are disadvantaged by being required to take Florida but has not passed the PP exam would be "incompetent," and, thus, potentially subject to disciplinary action. an exam that covers a broad range of areas beyond his or her area of specialty. Petitioner further contends that it is irrational to test an examinee on particular areas that are irrelevant to his or her work and/or desired career path. However, the PP exam for a particular discipline is specifically designed to ensure that a licensed P.E. is competent to practice over a range of specific areas encompassed within that particular discipline. This is because once a person becomes a licensed P.E., he or she may practice engineering within any discipline or specific area within that particular discipline, subject to the professional and ethical requirements to limit practice to the disciplines and areas in which the engineer is actually competent. Thus, the breadth of the PP exam is designed to help ensure minimal competence to practice engineering in a manner that protects the public health and safety. Petitioner also contends that because the PP exam for some engineering disciplines tests a broader range of areas than the PP exam may test for other disciplines, the exam inconsistently measures minimal competency across the range of engineering disciplines. However, as discussed in detail below, the subject matters tested on the PP exam for a given discipline are chosen by subject-matter experts who are licensed engineers practicing in that particular engineering discipline, and are deemed, by those subject matter experts, to be most important to test for purposes of measuring competency in that discipline. Thus, while the number of discrete subject matters tested on the PP exams may differ across the various engineering disciplines, this difference is, factually and logically, a function of expert consensus regarding which subject matters need to tested to demonstrate minimal competence. Petitioner also contends that the Challenged Rule is vague because it does not specifically identify the disciplines, and the areas within each discipline, that are tested on the PP exam. Respondent has contracted with NCEES to be responsible for preparing, administering, and scoring the PP exams, pursuant to section 455.217(1)(d). NCEES conducts a methodical process, discussed in detail below, to determine the specific disciplines for which to develop a PP exam and the areas to be covered on the PP exam for a discipline. Exam specifications are then developed by subject matter experts within that discipline, and are published by NCEES. These specifications inform prospective examinees regarding the particular areas that will be tested on the PP exam for the discipline, and the number of questions for each specific area that will appear on the exam. Thus, prospective examinees are not left to wonder or guess about which disciplines will be tested; the areas within each discipline that will be tested; or the relative weight that will be assigned to each area tested.12 Although the Challenged Rule does not identify the specific disciplines tested on the PP exam, rule 61G15-21.001(1)(b)—which actually adopts the PP exam as an engineering licensure exam in Florida—states that the PP exam "is given by discipline." Therefore, even if section 455.217(1)(d) required the specific areas of competency to be identified by rule, such areas would have been identified in rule 61G15-21.001(1), rather than in the Challenged Rule.13 12 See Cole Vision v. Dep't of Bus. And Prof'l Reg., 688 So. 2d 404, 410 (Fla. 1st DCA 1997)(a rule is impermissibly vague if it is drafted in terms so vague that men of common intelligence must necessarily guess at its meaning or application. 13 Neither sections 455.217(1)(d) nor 471.015(1) specifically authorize or require Respondent to adopt rules identifying the general areas of competency tested on the PP exam. By contrast, exams developed by the agency pursuant to section 455.217(1)(b) must identify, by rule, the general areas of competency to be tested. Had the Legislature intended for exams authorized under section 455.217(1)(d)—of which the PP exam is an example—to adhere to the same requirement, the statute would have so stated. See Pro-Art Dental Lab, Inc. v. V- Strategic Grp., LLC, 986 So. 2d 1244, 1258 (Fla. 2008)(the specific mention of one thing in a statute implies the exclusion of another). Furthermore, section 120.54(1)(g) expressly requires a rule to address only one subject. Thus, if the Challenged Rule also addressed the areas of competency to be tested on the PP exam, it would violate section 120.54(1)(g). Petitioner also asserted, at the final hearing, that Respondent did not certify the PP exam as meeting the requirements of national examinations and generally accepted testing standards pursuant to department rules, as required by section 455.217(1)(d). However, Petitioner did not raise this alleged invalidity basis in the Rule Challenge Petition, so he is foreclosed from raising and litigating it at the final hearing. See § 120.56(1)(b), Fla. Stat. Petitioner also testified that, in transitioning from paper-and-pencil PP exams to computer-based exams, NCEES is relying on two different theories—classical test theory and item response theory—and that this reliance does not comport with generally accepted testing standards. However, as discussed above, Petitioner was not qualified as an expert in the specialized area of high-stakes examination preparation and scoring; thus, his testimony constitutes lay opinion regarding this specialized subject matter. He did not present any competent substantial evidence to support his contention that the PP exam does not meet generally accepted testing standards.14 Petitioner also testified that item response theory, which is a psychometric tool used in developing and scoring the PP exam, is an invalid means of determining the competence of an engineer. To this point, Petitioner testified that the "model law engineer" is not a real person, but is instead an imaginary person created by subject matter experts to define what a minimally competent engineer should know. Thus, according to Petitioner, 14 Because Petitioner was not qualified, tendered, or accepted as an expert in these specialized subject matters, and because his testimony on these matters consists of inadmissible lay opinion testimony, this testimony has not been afforded weight. §§ 90.701 and 90.702, Fla. Stat. the model law engineer standard is the wrong reference point for determining minimal competency to practice engineering.15 Petitioner also testified that the PP exam does not accurately measure ability, which is a latent trait for which an arbitrary measurement scale must be created. He testified that the model law engineer standard is the midpoint of this scale, and that the purpose of the scale is to determine whether examinees fall above or below that midpoint.16 He further contended that the PP exam does not accurately measure ability, because performance on the exam may be influenced by extraneous variables, such as test anxiety. Petitioner also testified regarding item response theory, which, as previously noted, is a psychometric tool used in developing and scoring high- stakes exams—a subject about which Petitioner had no training in, or knowledge about, until he prepared for the final hearing in this proceeding. Specifically, Petitioner testified that the item characteristic curve is the basic building block of item response theory, and that there are two technical properties of an item characteristic curve: difficulty of the item, and the ability of the item to discriminate between examinees' abilities. Petitioner testified that another basic principle of item response theory is that the examinee's ability is a variable with respect to the items used to determine it. According to Petitioner, this principle rests on two 15 Petitioner's testimony on this point was based on excerpts from a book titled Item Response Theory and a book titled The End of Average. The Item Response Theory book is a treatise on psychometrics, a highly specialized field about which Petitioner was not qualified to testify as an expert, and which is not susceptible to lay witness opinion testimony. Thus, Petitioner's testimony on these points is not assigned weight. See §§ 90.701 and 90.702, Fla. Stat. Additionally, excerpts from The End of Average were determined irrelevant, so were not admitted into evidence. 16 As support for this testimony, Petitioner selectively cited and quoted the deposition testimony of Timothy Miller, Respondent's expert on the development and scoring of NCEES's psychometric-based PP exams. The specific context of Miller's deposition testimony was that when an exam item is overexposed, it is subject to drift, which means that the percentage of correct answers for the item increases to the point that the item no longer is a good discriminator. As further discussed below, Petitioner's testimony on this point was directly and persuasively countered by Miller's expert testimony regarding scaled scores and setting the passing score for the PP exam. conditions: that all items measure the same underlying latent trait, and the values of all item parameters are in a common metric. According to Petitioner, this principle reflects that the item characteristic curve spans the entire ability scale; thus, the practical implication is that a test located anywhere along the ability scale can be used to estimate an examinee's ability, such that an examinee could take a test that is easy or hard, and on average, would score at the same estimated ability level. Petitioner testified that this stands in contrast to classical test theory, which he contends is a better discriminator of examinee ability.17 Petitioner also testified that the psychometric methods used to develop and score the computer-based PP exams are flawed because "difficulty" is subjective and entirely dependent on the individuals developing the PP exam. Thus, according to Petitioner, in scoring a computer-based multiple-choice PP exam, it is impossible to know whether a particular examinee got the answer right due to a reasonable approach in answering the question, or by guessing. Petitioner contends that for this reason, multiple-choice test questions developed using item response theory are not good discriminators of examinees' ability; thus, even if an examinee does not correctly answer enough questions to pass the exam, that does not mean that the examinee is not knowledgeable in that area.18 Petitioner further testified that because difficulty is a subjective parameter, different forms of the PP exam inherently have different levels of difficulty. Thus, according to Petitioner, it is a matter of luck whether an examinee takes a more difficult form or an easier form of the exam. Further to this point, Petitioner testified that because an examinee does not take multiple forms of the exam, but instead takes only one form, the determination of the examinee's ability is solely dependent on a subjective 17 Refer to note 15, supra. 18 Refer to note 15, supra. parameter—i.e., the difficulty of the test questions as determined by subject matter experts. Petitioner contends that, as a result, the PP exam does not accurately measure an examinee's ability, and, therefore, is not a valid exam.19 Petitioner also testified that because statistical indices of reliability and validity are not attributes of an exam, a researcher may select what seems to be an appropriate test for his or her purposes, when, in fact, the selected test does not have any level of reliability or validity. Thus, Petitioner testified, reliability and validity are values that reside in test scoring, not in the test itself. Petitioner testified that validation, in statistics, is the process of accumulating evidence that supports the appropriateness of the inferences that are made of student responses for assessment uses. He testified that validity refers to the degree to which the evidence indicates these interpretations are correct and the manner in which the interpretations are used is appropriate.20 Petitioner testified regarding three types of validity evidence: content, construct, and criterion evidence.21 Specifically, Petitioner testified that content evidence refers to the extent to which an examinee's responses to a given assessment reflect the examinee's knowledge of the content being tested; thus, to the extent an exam inadvertently measures a parameter that is not related to the examinee's knowledge of the content being tested, it is invalid. 19 Refer to note 15, supra. 20 Petitioner's testimony relied on, or was paraphrased from, a document titled The Scoring Rubric Development. Again, because this topic and document address a matter within the specialized fields of psychometrics, high-stakes testing, and test-scoring statistics, which are areas in which Petitioner was not qualified as an expert, and which are not susceptible to lay opinion testimony, pursuant to sections 90.701 and 90.702, Petitioner's testimony relying on this document, including his testimony regarding content, construct, and criterion-related evidence, is not afforded weight. 21 Refer to note 15, supra. This determination regarding the weight afforded Petitioner's testimony applies to paragraphs 94 through 100 herein. According to Petitioner, the content-related evidence for the PP exam for each discipline is inconsistent, so that the PP exam for a given discipline does not accurately measure minimal competence for that discipline. Petitioner also testified that the weighting of different topics on the PP exam necessarily creates an advantage for engineers who work in areas more heavily weighted on the exam, while creating a disadvantage for engineers who work in areas that are less heavily weighted on the exam. Petitioner also testified that to accurately determine minimal competence in all engineers, the model law engineer standard should be keyed to, and the content tested on the exam should be directed toward all engineers, including those having more than four years of experience. Petitioner also testified that construct-related evidence consists of external benchmarks, such as results and explanations, of internal evidence of psychological processes, such as reasoning. Petitioner testified that because multiple-choice exams do not provide evidence of an individual's reasoning process, they do not generate construct-related evidence for purposes of determining exam validity. Petitioner testified that free response paper-and pencil-exams provide construct-related evidence, so are better indicators of an examinee's knowledge. Petitioner also testified regarding criterion-related evidence, which relates to the extent to which the results of an assessment, such as the PP exam, correlate with a current or future event. By way of illustration, Petitioner testified that criterion-related evidence considers the extent to which a student's performance may be generalized to other relevant areas. Petitioner testified that an examinee's performance on the PP exam is not generalizable to other relevant activities, so it is impossible to determine whether the exam actually corresponds to minimal competence in the workplace. In sum, Petitioner testified that the PP exam does not meet content, construct, or criterion-based evidence for purposes of determining whether it is a valid exam. Thus, Petitioner reasons, it is logical to conclude that because the PP exam is not a good discriminator between minimally competent and incompetent engineers, it does not reliably and adequately measure competence. Petitioner also testified that because passing the PP exam is only one component of licensure, it fails to meet criterion-based validity, in that the exam, by itself, does not certify a passing examinee to practice as a P.E. As Petitioner put it, "you're just passing the exam as part of the requirement for licensure." Petitioner reasoned that if passing the PP exam corresponds to minimal competence, the experience and education requirements of section 471.015(2)(a) are redundant. Also to this point, Petitioner testified that the preapproval process to take the PP exam is directly related to an examinee's actual work experience as an engineer, while taking the exam merely entails answering questions about engineering work. Thus, Petitioner contends, a competent engineer, as shown through Respondent's preapproval process, may nonetheless fail the exam. Petitioner asserts that this further shows that the exam does not accurately measure minimal competence. Petitioner also testified that, in his view, delaying licensure of potentially competent engineers due to postponing the exam due to the Covid-19 pandemic does not serve the interest of public health and safety. To that point, he testified that the inability to obtain a variance, which would relieve examinees from having to take and pass the exam under such circumstances, renders the Challenged Rule arbitrary.22 Petitioner also contended that passing the PP exam should not be required, because other engineering professional associations—specifically, the European Federation of National Engineering Associations ("FEANI"), which represents engineers in European countries—allow licensure through 22 Petitioner's point regarding inability to obtain a variance or waiver is addressed in the Conclusions of Law. education and experience requirement, without requiring a professional exam to be taken and passed. However, because section 471.015, which governs the licensure of engineers in the state of Florida, requires a professional licensure examination to be taken and passed as part of the P.E. licensure requirements, Petitioner's testimony and argument regarding FEANI's practices and requirements take issue with the statute, rather than the Challenged Rule, and, thus are irrelevant. Petitioner also contends that the examination fee for the PP exam is arbitrarily set, rendering the Challenged Rule arbitrary. However, as discussed above, the Challenged Rule only addresses determining the passing grade for the PP exam using psychometric methods. The Challenged Rule has nothing to do with establishing or setting an examination fee. Thus, this challenge ground has no basis in fact or law.23 C. Findings Regarding the Evidence Presented in Respondent's Case Respondent's Expert Witnesses Respondent presented the testimony of Timothy Miller, P.E., who serves as Director of Examination Services for NCEES. Miller has held this position for approximately 15 years. His job-related activities and responsibilities include directing exam development, publication, scoring, and fulfillment of the licensing exams for engineers and surveyors; coordinating exam development committees consisting of over 800 volunteer subject matter experts who work on developing each NCEES exam; overseeing the exam development process and providing advice and guidance regarding engineering exam development, administration, production, scoring, analysis, and reporting; serving as a testing process consultant to exam development committees; and other exam-development and administration-related matters. 23 Additionally, this challenge ground was not raised in the Rule Challenge Petition, so is not at issue in this proceeding. See § 120.56(1)(b), Fla. Stat. Before Miller was promoted to his current position, he served as an examination development engineer for NCEES. In that position, Miller was responsible for planning and coordinating engineering exam development, production, administration, scoring analysis, and reporting for certain assigned examinations; serving as a testing consultant working with engineering exam development committee chairs regarding quality and number of exam development volunteers; and overseeing development and administration of the licensing exams in the specific fields of environmental controls systems, metallurgical engineering, and mechanical engineering. Through his experience in these positions with NCEES, Miller is an expert in professional examination development and scoring, particularly with respect to the development and scoring of the NCEES FE and PP examinations. Before being employed with NCEES, Miller practiced civil and structural engineering with several private-sector engineering firms. He has been a professional engineer since 1984, and is licensed as a P.E. in South Carolina, North Carolina, Maryland, Delaware, Pennsylvania, and New Jersey. Respondent also presented the testimony of Dr. Michelle Rodenberry, P.E., an associate dean and associate professor at the Florida A&M University–Florida State University College of Engineering. Her engineering expertise is in the field of structural engineering—specifically, bridge engineering. Rodenberry was appointed to the Board in 2012, and she served as a Board member until 2018. She is now an emeritus Board member. While on the Board, she served as chair of the education committee, and was involved in reviewing applications for licensure as a P.E. in Florida. Development, Scoring, and Validation of the PP Exam The NCEES engineering exams are national licensing exams that are recognized by every engineering licensing entity in each of the U.S. states, as well as by the engineering licensing entities in Washington, D.C.; Puerto Rico; the U.S. Virgin Islands; and the other U.S. territories and protectorates. There are approximately 26 different engineering disciplines, each of which is tested by a separate PP exam specific to that discipline. In the 1990s, NCEES decided to transition from subjectively-graded pencil-and-paper examinations to an objectively-graded computer-based multiple-choice exam format. Currently, approximately one-third of the PP exams, including environmental engineering, have been converted to a computer-based format, and all but one of the exams in the other disciplines is in the process of being converted. The reason NCEES is transitioning the PP exam from a pencil-and- paper format to a computer-based format consisting of multiple-choice questions is to help eliminate subjectivity in grading, so that the exam papers are consistently graded across groups of examinees. Additionally, a computer-based format consisting of objective multiple-choice questions allows the difficulty of the exam to be psychometrically evaluated for purposes of determining the passing score for a particular administration of the exam. To that point, because computer- based multiple-choice exams are objectively scored, exams offered at different times during the year are able to be compared, or equated, for purposes of setting the passing grade for a particular exam administration.24 Respondent entered into a contract with NCEES in 2009, pursuant to which NCEES provides the FE and PP exams for engineer licensure in Florida. In 2013, the contract was amended to allow NCEES to provide the exams by computer-based testing, using Pearson Vue as its exam 24 As Miller explained,"[i]f they were different on a difficulty level, the harder exam, the standard would actually be lowered so that it would be fair across administration so everybody was treated consistently. Or if my exam was less difficult, the standard would be raised. I would have to get more questions right." administering entity. The FE and PP exams are administered by Pearson Vue at its testing centers. NCEES develops model laws and rules that represent best practices with respect to state licensing of engineers. The aim of these model laws and rules is to achieve uniformity and consistency throughout the states and the U.S. territories and protectorates in the licensure of professional engineers. A significant benefit of such consistency and uniformity is the resulting "mobility" for licensed professional engineers—that is, the ability to more easily become licensed to practice engineering in multiple states. The NCEES model laws and rules establish the "model law engineer," which defines and constitutes the standard for minimal competence in a specific engineering discipline for purposes of being licensed as a P.E. in that discipline. The model laws and rules define the "model law engineer" as a person who holds a degree from an engineering educational program accredited by ABET, has four years of active engineering practice experience, and passes the FE and PP exams. The model law engineer standard equates to the competence level of an engineer having four years of engineering experience who is capable of practicing engineering in a manner that protects the public health and safety. This constitutes the minimum competence level that an applicant must demonstrate for purposes of being licensed as a P.E.25 in the 50 states and the U.S. territories and protectorates. Thus, the NCEES PP exam is constructed to test engineering ability keyed to the model law engineer standard. That is, the PP exam is designed to determine the ability level of an applicant for P.E. licensure for purposes of 25 Refer to note 12, supra. The term "minimal competence," as used in the Challenged Rule, is specifically keyed to the "model law engineer" standard for purposes of being licensed as a P.E. It is not meant to indicate or imply that an engineer who does not take or pass the PP exam is per se incompetent, such that he or she is not competent to engage in work constituting engineering, as defined in section 471.005(7). comparing that ability level to that of an engineer having four years' experience who is able to practice engineering in a manner that protects the public health and safety. Examinees having four years of engineering experience after graduation have the highest pass rates on the PP exams. Pass rates for examinees with more or less than four years of experience are lower, typically in proportion to the length of time before or after the four-year experience mark when they take the PP exam. Miller explained, credibly and persuasively, that the reason for the drop-off in PP exam performance after the four-year mark is that "life happens." Engineers gain more experience, and many become specialized in a relatively narrow niche, or move into managerial, non-technical positions. Additionally, because the PP exam does, in part, test subjects that one learns in college, the longer an examinee is out of college, the less subject matter recall in certain areas he or she may have. "Psychometrics" is the specialized field of study concerned with the theory and technique of psychological measurement. Specifically, psychometrics entails the objective measurement of skills and knowledge, abilities, and educational achievement. Among other specialized areas of practice, psychometricians focus on the construction and validation of assessment instruments, and theories, such as item response theory, that relate to psychological measurement. Psychometricians typically have graduate training and all possess specialized qualifications that enable them to engage in objective psychological measurement. PP exams are designed to determine minimal competence in a specific engineering discipline. "Minimal competence" is the minimal amount of knowledge required to practice in that particular engineering discipline in order to protect the public health and safety. For any specific engineering discipline for which it has been determined that a PP exam should be given,26 there is an approximately three-year due diligence period in which subject matter experts in that discipline work to determine the topics that should be tested on the exam. The PP exam for each specific discipline is developed by subject matter experts, who volunteer and meet on a monthly basis to develop, review, and evaluate the questions for the PP exam for that specific discipline.27 The process of determining which topics should be tested on a PP exam, termed the Professional Activities and Knowledge Study ("PAKS") process, is a standard practice used to determine the specific topics to be tested on a PP exam. As part of the PAKS process, a consulting psychometrician28 employed by Pearson Vue; 15 to 20 engineers who are licensed in another engineering discipline; and subject matter experts who may teach a particular engineering discipline for which the PP exam is being developed, work together to develop consensus regarding the specific topics that engineers having four years of experience practicing in that discipline need to know in order to safely practice engineering in a manner that protects the public health and safety.29 26 For a PP exam to be developed for a new engineering discipline, at least ten NCEES- member state engineering boards must request that such exam be developed, and at least one ABET-accredited program in that specific discipline must exist. 27 Over the years of development and administration of the PP exam, hundreds of licensed engineers have provided input regarding the topics that should be, and are, tested in each discipline and the relative weight given to each topic on the PP exam. 28 Pearson Vue's psychometricians who develop, score, and evaluate NCEES's exams have Ph.D. degrees in psychometrics or statistical analysis. 29 Subject matter experts selected to develop the PP exam questions are chosen based on consideration of the type of practice, such as governmental and private practice; gender; ethnicity; length of time of licensure as a P.E.; and geographic considerations. All subject matter experts must be licensed as a P.E. by a state engineering licensure board in order to The consulting psychometrician builds a questionnaire that lists the specific topics identified by the PAKS committee, and distributes an online survey to engineers who practice in the discipline for which the PP exam is being developed. The survey seeks input regarding the relative importance of each specific topic for purposes of testing to demonstrate minimal competence in the discipline. Based on the survey responses from engineers practicing in the discipline, exam specifications are developed. The exam specifications identify each specific topic to be tested on the PP exam, and the number or percentage of exam questions that will address each specific topic within that discipline. The exam specifications must be approved by an oversight committee. Once the exam specifications have been approved, the subject matter experts for that specific engineering discipline for which the PP exam is being developed prepare the PP exam questions—also termed "items"—and review and evaluate them for clarity, demographic neutrality, and other parameters, so that the items will reliably and validly test engineering ability. In computer-based multiple-choice PP exams, the questions are prepared such that for each question, there is only one correct answer and three other plausible, but incorrect, alternative choices. The individual exam questions are reviewed numerous times by the subject matter experts before they are moved into an exam question bank for use on the PP exam. Once the exam questions have been developed and banked for use on a PP exam, a standard-setting committee, consisting of ten to 15 licensed engineers having diverse backgrounds, experience, and demographic features, reviews the exam to determine the minimum passing score—or "cut score"—on the exam. The cut score equates to the ability level of an engineer serve in this capacity. As noted above, over the years of development and administration of the PP exam, hundreds of engineers have provided input in developing each PP exam. having four years of experience who is minimally competent to practice engineering at a level that protects the public health and safety. This method of using subject matter experts to examine the content of each exam question and predict how many minimally-qualified examinees would answer each question correctly is termed the "Modified Angoff Method." The standard-setting committee then takes the exam, and the psychometrician analyzes the data from the committee's exam sitting. Using this data and analysis, the standard-setting committee then reviews, and reaches consensus, regarding each question, for purposes of determining the proportion of minimally competent engineers who would answer that specific question correctly. Based on the information generated by this process, the psychometrician develops the "panel recommended passing score," with a statistical margin of error. The psychometrician presents this recommended passing score to a committee of five persons, consisting of two state licensing board members and three subject matter experts who observed the exam development process. Based on the psychometrician's recommendation, the committee makes the final decision regarding the minimum passing score for the exam. Each PP exam question is developed and evaluated using the process described above, and is placed in bank for use on a PP exam. The psychometrician uses item response theory to "calibrate"—i.e., determine the relative difficulty level of—each exam question.30 An exam question is not banked for use on future sittings of the exam unless it has had at least 200 responses on a previous exam, so that statistics for each item's performance can be generated for purposes of item calibration. 30 Item response theory is one of many psychometric methods, or tools, used to weight exam questions for purposes of creating different forms of exams having the same level of difficulty. Depending on the specific discipline, a question bank for a PP exam may consist of many thousands of questions.31 Using item response theory, the psychometrician converts the passing score to create a scale from -5 to +5, which will equate to the examinee's ability level as measured by the exam. Once the passing score for the PP exam has been determined, different PP exam "forms" are created for administration in different exam sittings. Exam "forms" are essentially different versions of the PP exam that consist of different individual questions of the same difficulty level, as determined using item response theory, for each specific topic on the exam. Thus, if a PP exam was administered, for example, in April and October, the different exam administrations would consist of different forms—meaning that the exam would consist of different questions, but the questions would be of the same difficulty level for a specific topic tested on the exam.32 Additionally, because exam item difficulty has been calibrated using item response theory, different forms of a PP exam can be given during the same exam administration at different locations.33 Importantly, because the difficulty of the exam items has been calibrated using item response theory, the different exam forms are statistically equivalent in difficulty. 31 The only items that will be used on the graded portion of the PP exam are questions that have known statistics such that they have been calibrated for difficulty. However, there may be other "pretest" questions on the exam strictly for purposes of gathering statistics regarding performance on the questions for potential inclusion in the exam item bank; these "pretest" exam questions are not graded for purposes of determining the examinee's score on the exam. 32 As Miller explained, for an administration of an exam at different locations at the same time, the form administered at a particular location consists of different questions than the form administered at another location; however, the exam forms are equivalent in terms of the number of questions addressing a particular topic and the difficulty of the items addressing that topic. 33 Using the "linear-on-the-fly" ("LOFT") method to generate different forms of the exam also helps ensures exam security, since persons sitting near each other during an administration of the exam will not have the same exam form. As a result of using these processes, including the Modified Angoff Method, and applying item response theory to calibrate the exam items for purposes of constructing different, but statistically equivalent, forms of the PP exam, examinees are not graded on how they perform against each other, but instead are graded against the cut score set for the exam. To ensure that different forms of the exam are statistically equivalent in difficulty, Pearson Vue uses the LOFT method,34 which employs an algorithm to ensure that, across all of the exam forms, all examinees get the same number of questions of the same level of difficulty on the same topics. The algorithm randomly generates, or assembles from banked exam questions, different exam forms based on the exam specifications (i.e., the specific topics tested and relative weight/number of exam questions for that topic) and the difficulty level of the exam questions, such that the different exam forms generated by LOFT are statistically equivalent to each other. Using item response theory to calibrate specific exam question difficulty based on the statistical probability of being answered correctly enables examinees taking different, but statistically equivalent, forms of the exam to be compared to the passing standard for purposes of determining whether they pass the exam. Thus, examinees are compared to an ability level—here, minimal competence—rather than to each other.35 This method ensures that all examinees take an exam of equivalent difficulty, which, in turn, helps ensure the fairness of the exam. 34 For the engineering disciplines having too small a group of examinees to employ item response theory or LOFT to generate different exam forms, each examinee takes the same exam instead of taking different forms of the exam, and the exam typically is offered only on one day, rather than multiple days, per year. 35 By way of example, Miller explained that if two examinees each answer five questions on the same topic on the exam, and one examinee answers four easier questions correctly and the other examinee answers two comparatively more difficult questions correctly, the examinee answering the two more difficult questions correctly may have a higher ability level on that particular topic, due to the comparative difficulty of the questions that examinee answered correctly. Once a PP exam is administered, Pearson Vue scores the examination and sends NCEES the information regarding whether each examinee has passed or failed the exam. Pearson Vue also provides each examinee the information regarding his or her performance on the exam compared to the minimum competence standard. The examinee's performance is expressed as a scaled score, for each specific topic tested on the exam, and for the entire exam. Specifically, using psychometric statistical methods, the ability level of the examinee is expressed as "theta," and is placed on a scale of 0 to 15 for each of the specific topics tested. The examinee's overall theta across all specific topics tested is then compared to the "minimal competence" passing standard, which is also expressed as a scaled score using the same 0 to 15 scale. After an exam is taken and scored, the consulting psychometrician analyzes this data, called "response data," for each exam question, for each examinee, to calibrate the items for purposes of determining whether a particular question performs well in discriminating ability level of the examinees. The psychometrician may recommend that an exam question be "retired" because it is not performing as a good discriminator of ability level. Examples are where an exam question is too difficult or too easy, such that it does not discriminate well in determining ability level; where an item takes too long to answer or is ambiguous; where an item has been "overexposed" by having become publicized such that future examinees have access to the question and scores on the question become high; or where an exam contains "bad pair" items, such that the answer to one item may suggest, or lead to, the answer on another similar item. Having a psychometrician involved in tracking and analyzing exam data enables such circumstances and situations—which may influence the scores on a test item for reasons not related to the examinee's ability—to be identified and corrected. Returning to a free response, paper-and-pencil exam format for the P.E. licensing exam would provide a far less objective, fair, and accurate measure for determining minimal competence for purposes of being licensed to practice engineering. It also would negatively affect the ability of licensed engineers to become licensed in other states. Due to the use of psychometric methods in developing and scoring, the PP exams are very reliable across multiple administrations of the exam— to the point that all of NCEES's psychometric-based PP exams score upwards of .9 on a scale of 1.0.36 Psychometric methods, including item response theory, are used in developing, administering, and scoring many different types of high-stakes professional and academic examinations, including medical school admissions examinations, and nursing, medical examiner, internal auditor, and architecture licensure examinations. NCEES audits approximately one-third of the exams administered by Pearson Vue on an annual basis, to independently evaluate the accuracy of the psychometric services provided by Pearson Vue, and to ensure that the exams have been created pursuant to NCEES's guidelines, procedures, and requirements. NCEES also retains independent psychometricians to review Pearson Vue's exam-related reports and analyses, to ensure that Pearson Vue is following standard psychometric rules of good practice. In sum, the use of objective psychometric methods, including the methods discussed above, to develop, score, and evaluate the PP exam ensures that minimal competence, for purposes of licensure as a P.E., is accurately measured by the exam. Role of the PP Exam in Licensure of PEs in Florida As discussed above, to be licensed as a P.E. in Florida, an applicant must have a college degree from an ABET-accredited institution, four years of 36 Test reliability refers to the degree of consistency with which a test measures a particular subject matter across different administrations of the test. A test has a high reliability score if it consistently produces similar results under consistent conditions. A 1.0 reliability score reflects perfect consistency in results across different administrations of a test. An acceptable reliability score target for high-stakes tests is .7 or higher. active experience in engineering practice, and have passed the FE and PP exams. Thus, the P.E. exam is a vital component of determining that an engineer licensed as a P.E. to practice in Florida is able to practice at a competence level that protects the public health and safety. Unlike the education and experience requirements for licensure— both of which may entail a great deal of variability in quality and breadth across applicants—the PP exam constitutes an objective, consistent tool37 to measure an applicant's level of competence for purposes of determining whether the applicant possesses the minimal competence needed to practice engineering in a manner that protects the public health and safety. As such, the PP exam constitutes a uniform measure of minimal competency for purposes of licensure as a P.E. in Florida. As discussed above, this does not mean that a person who engages in engineering work but has not passed the PP exam is incompetent; it merely means that he or she has not demonstrated minimal competency on this required objective measure of competency for licensure purposes. As discussed above, the PP exam is specifically designed to ensure that a licensed P.E. is competent to practice over a range of specific areas encompassed within a particular discipline. This is because a licensed P.E. is authorized to practice engineering within any discipline or area, subject to professional and ethical standards. The breadth of the PP exam thus helps ensure minimal competence to practice engineering in a manner that protects the public health and safety. 37 This consistency and uniformity is the direct result of the psychometrically-based exam development, scoring, calibration, and validation methods discussed above. The purpose of P.E. licensure is to inform and protect the public, which is entitled to rely on such licensure as indicating that the licensee is competent to practice engineering.38 Administration of the PP Exam During the Covid-19 Pandemic As discussed above, Petitioner has alleged that the Challenged Rule is arbitrary on the basis that it does not address contingencies for offering the exam if unforeseen circumstances prevent regular administration of the PP exam. Specifically, Petitioner points to the fact that Pearson Vue cancelled the April 2020 PP exam administration due to the Covid-19 pandemic. Pearson Vue has rescheduled the PP exams for various times and at various locations around the country in an effort to make the PP exam available for prospective examinees during the pandemic.39 Pearson Vue also is taking substantial steps to protect persons who have applied to take the PP exams during the Covid-19 pandemic. To that point, Pearson Vue has retrofitted its testing centers to help ensure the safety of the examinees as they sit for the PP exam. Specifically, the number of examinees who will be in a testing room has been reduced; masks are required to be worn by examinees and proctors; testing stations are cleaned between each use; some additional test center locations have been added; and some states have relaxed rules to allow examinees to sit for the exam in states other than the one for which they are applying for licensure. At present, the exams are not able to be offered over the internet so that examinees are able to take the exam at a remote location. A substantial reason for this is lack of exam security, which is necessary to protect and 38 As noted above, a person does not have to be licensed as a P.E. to engage in engineering work in Florida. However, if a person wishes to hold himself or herself out to the public as a P.E., then that person must satisfy the requirement to pass the PP exam, which is an indicator of minimal competence for purposes of licensure. 39 Among other things, Petitioner alleges, in paragraph 19 of the Rule Challenge Petition, that the Challenged Rule is invalid under section 120.52(8)(d) because it does not address circumstances where an examination cannot be administered due to force majeure. As maintain the exam's integrity. Additionally, the lack of guarantee of internet service reliability and functionality for every examinee is a crucial consideration, since failed internet connections could significantly affect the fairness of the exam.
Findings Of Fact The Petitioner, Glen P. Hamner, Jr., has applied for licensure by examination to practice architecture in the State of Florida. The architectural licensure examination, administered by the Respondent, consists of two portions, the written examination given in December of each year, and the site planning and design portion administered in June of each year. The Petitioner has complied with all requirements for admittance to the subject examination. The Petitioner sat for the 12-hour "Part A" examination in June, 1980. The examination consists of a drafting or sketching problem and is so constituted as to require the applicant for licensure to design a particular type of building to be accommodated to a particular site, including requirements for placing the structure on the site, designing elevations, building cross sections, facades and floor plans, as well as taking into consideration numerous criteria such as human traffic flow, parking, access to all areas, heating and cooling, including solar heating potential, prevailing climate conditions, use of natural lighting, and numerous other esthetic, engineering and legal requirements. The examination is administered by the Office of Examination Services of the Department of Professional Regulation and is supplied to the State of Florida, as well as to all other jurisdictions in the United States by the National Council of Architectural Registration Boards (NCARB). The examination was adopted in Florida pursuant to the above-cited rules. Prior to sitting for the examination, each applicant, including the Petitioner, receives a pre-examination booklet setting forth the architectural program to be accomplished by that applicant and various requirements to which the Petitioner was expected to apply himself in order to receive a passing grade. Immediately prior to commencing the examination itself, the Petitioner received other information designed to enable him to more adequately design the structure requested and perform the necessary technical and architectural requirements of the problem. In general, the examination was designed to require the Petitioner to design a solution to the site plan and building design problem submitted to him by the NCARB and the Florida board. The pertinent portion of the examination thus allows the examination graders and, through them, the Florida Board of Architecture to determine whether an applicant, such as the Petitioner, is able to coordinate the various structural, design, technical, esthetic, energy and legal requirements in order to resolve the design and site plan problem after having been tested on the same requirements in written form in the initial portion of the examination administered in December of each year. The grading of the design and site portion of the examination was accomplished by submission of the Petitioner's work product to at least three architects selected by the various architectural registration boards of 20 states. These graders are given training by the NCARB in order to standardize their conceptions of minimal competence required for achievement of a satisfactory grade on the examination. Each architect grader is then asked to review and score various solutions to this site and design problem, including the Petitioner's, in a blind grading basis. The grader has no knowledge of the name or state of origin of the applicant whose solution he is grading. The grader is instructed to take into consideration the various criteria set forth in Rule 21B-14.03, Florida Administrative Code, as well as in Respondent's Exhibit Three. The graders are instructed to note the areas of strength and weakness in an applicant's solution with regard to those grading criteria and then determine, based on an overall conception of the solution submitted by the applicant, whether or not a passing grade is warranted. A passing grade is defined as a holistic grade of three or four as set forth in Rule 21B-14.04, Florida Administrative Code. The applicant must receive at least two passing grades from the three architect graders who independently grade his solution in order to pass that portion of the examination. The Petitioner herein received two "1's" and one "2" on the examination, all of which were failing grades. Although he demonstrated an effort to comply with instructions set forth in the examination, as well as the pre-examination booklet, he failed to achieve sufficient clarity of presentation in several material areas such that the graders could make a clear determination that he understood and had complied with sufficient of the mandatory criteria to achieve passage of the examination. The testimony of the Respondent's witnesses (Herbert Coons, Executive Director of the Florida Board of Architecture and a grader in this examination, and Mr. Dan Branch, a grader of the petitioner's own examination) shows that the petitioner failed to supply sufficient information to permit a passing score to be awarded based on the criteria required to be considered and complied with by the authority cited below. The Petitioner's examination was deficient in a number of material respects. Many technical errors were pointed out by these witnesses, as well as by the petitioner's own admissions. The record thus discloses that the Petitioner's examination solution was deficient in its allowance for parking space and the ability of vehicular ingress to egress from the parking lot surrounding the building to be designed. There were no room designations on the rooms in the floor plan; there was only one rest room space and one set of toilets for men and women where two spaces and sets of facilities were required by the instruction program. The Petitioner failed to mark the building elevations on the building floor plans, did not depict where furniture would be located, and two different elevation drawings and two sectional drawings required to be depicted on the examination were not. The Petitioner did not show what type of material would be used in the roof nor what type of insulation, nor did he show what type and uses of glass were to be made. The Petitioner failed to give adequate consideration to grading and site planning, failed to adequately make notation of the types of materials to be used in the elevation's floor plans and wall sections, and generally did not adequately adhere to the program presented him. In general, it was shown by these witnesses that while the Petitioner had made a substantial effort to pass the examination, he had failed to place within the solution adequate information to allow the graders to determine clearly that his program or design could be a successful one. The Petitioner's own admissions show that he approached the examination in question in such a manner as to substitute his own judgment and opinion regarding which techniques and components were architecturally sound for the problem for those required to be treated as essential elements of the site and design problem posed him in the instructions. The program presented to the Petitioner assumed he would be in the hypothetical position of an "architect" presented with a program which had already reached a point of completion as to design ideas and site location. The insertion of his own ideas and judgment regarding various elements of the project is contrary to established architectural practice, for a program which had already reached the point of completion, in terms of initial design decisions, as that presented to the examination candidates in this instance. In view of the above-determined deficiencies, the Petitioner did not establish that his solution to the site and design-problem posed by the examination reflected sufficient and appropriate consideration of the requirements and criteria he was instructed to address.
Recommendation Having considered the foregoing findings of fact and conclusions of law, the evidence in the record, the candor and demeanor of the witnesses, and the pleadings and arguments of the parties, it is RECOMMENDED that the failing grade conferred on the Petitioner on the June, 1980, site and design portion of the architectural examination be upheld, and that the petition be denied. DONE and ENTERED this 19th day of November, 1981, in Tallahassee, Florida. P. MICHAEL RUFF, Hearing Officer Division of Administrative Hearings The Oakland Building 2009 Apalachee Parkway Tallahassee, Florida 32301 (904) 488-9675 Filed with the Clerk of the Division of Administrative Hearings this 19th day of November, 1981. COPIES FURNISHED: Mr. Glen P. Hamner, Jr. 1231 Bayshore Drive Valparaiso, Florida 32580 John Rimes, Esquire Department of Legal Affairs The Capitol, Suite 1601 Tallahassee, Florida 32301 Samual Shorstein, Secretary Department of Professional Regulation 130 North Monroe Street Tallahassee, Florida 32301
The Issue Is Petitioner entitled to additional credit on the Professional Engineer Licensure Examination sufficient to receive a passing score?
Findings Of Fact By the stipulations recited in the preliminary statement, Petitioner needed only to demonstrate entitlement to one raw point in order to achieve an adjusted score of 70 (raw score of 48) so as to pass the Florida Professional Engineer Examination, created and administered by the National Council of Examiners for Engineers and Surveyors. The disputed part of the challenged question dealt with the equations necessary to calculate the amount of excess air applied to a combustion source that produced flue gas which contained specified concentrations of oxygen, carbon dioxide, carbon monoxide, and nitrogen. This presents a chemical engineering problem. Subpart one of the question provided four reaction equations. Petitioner selected answer "E". The Board claims another answer is the correct answer. Subpart one of the question required that the examinee select an answer showing the minimum number of equations needed to solve the problem. The answer designated by the Board contained two equations. The answer selected by Petitioner contained three equations. Petitioner, Petitioner's Professional Engineer expert in combustion, and the Board's Professional Engineer expert in chemical engineering all recognized that the problem could be appropriately solved either by the Orsat method or the Board's preferred method. However, Petitioner and his expert maintained that the Board's preferred method was less precise than the Orsat method because the Board's method was based on a presumption of complete oxygen combustion or theoretical oxygen. Petitioner based his analysis on the concept that the problem's acknowledgment of the presence of carbon monoxide indicated incomplete combustion and rendered the calculation of air indeterminate. The determination of excess air is covered in five standard reference books authorized for use on the examination. All these books recognize the Orsat equation for the calculation of excess air with respect to incomplete combustion. Petitioner utilized the Orsat equation. The Orsat formula is appropriate for solving the question, but it is derived from three equations. Petitioner and his expert contended that subpart one of the question was both a trick question and that the Board was requiring an incorrect answer because the examinee obtained a correct answer to subpart ten of the question by the Orsat approach and that when subpart ten is solved by the Board's preferred method, an incorrect answer was obtained. This testimony was not persuasive in light of the apparent agreement of Petitioner that the correct numerical answer would be closer to the Board's preferred numerical answer than to his own numerical answer. (TR110112). Nor does it necessarily follow that because the Board has conceded that Petitioner's answer to subpart ten was correct, that concession automatically renders his answer to subpart one correct. As explained more reasonably by Dr. Benjamin Keith Harrison, a professor of chemical engineering at the University of Alabama, there were at least two correct ways to work the problem related to an analysis of incomplete combustion. The Orsat method uses carbon dioxide while the Board's preferred method uses a different chemical "tie" element to compare the theoretical oxygen required to the amount of oxygen actually in the system. The examinee is free to choose the method (i.e. tie element) he prefers. The Orsat formula, chosen by Petitioner, is entirely correct to do that, and the numerical answer the Petitioner got in subpart ten was within acceptable limits. However, the equations the Petitioner indicated in the other part of his answer were not those used to derive the Orsat formula. (TR8288) More succinctly, according to Dr. Harrison, the examinees were free to use either the Orsat formula or the Board's preferred formula to get subpart ten. Petitioner chose the Orsat formula and got a sufficiently correct answer for subpart ten. However, in subpart one, the other formula was asked for and Petitioner chose the wrong combination of three subsets. (TR89) Therefore, the greater weight of the credible evidence is that the Petitioner's answer is wrong on two points: first, that his response does not indicate a minimum set of equations and two, the three equations he selected were not the equations used in deriving the Orsat formula. Likewise, items one and ten of the question request different information/responses. Part ten requests a numerical response; part one requests the selection of the minimum number of equations necessary. Dr. Joseph Allen Klock, was accepted as an expert psychometrician. His statistics and the testimony of Dr. Harrison are credible and persuasive that the challenged question subpart one contained enough correct information to allow an examinee of minimal competency for licensure to selec tthe correct response and did not require knowledge which was beyond the scope of knowledge that could be expected from a candidate for licensure and that Question No. 417 as a whole was a fair test of the examinees' knowledge in the field of chemical engineering .
Recommendation Upon the foregoing findings of fact and conclusions of law, it is recommended that the Board of Professional Engineers enter a final order denying the Petitioner the one point at issue, and thus a passing grade on the April 1995 licensure examination. RECOMMENDED this 3rd day of October, 1996, at Tallahassee, Florida. ELLA JANE P. DAVIS Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (904) 4889675 SUNCOM 278-9675 Fax Filing (904) 9216847 Filed with the Clerk of the Division of Administrative Hearings this 3rd day of October, 1996. COPIES FURNISHED: William Leffler, III, Esquire 2000 North Meridian Road Apartment 312 Tallahassee, Florida 32303 R. Beth Atchison, Esquire Department of Business and Professional Regulation Board of Professional Engineers 1940 North Monroe Street Tallahassee, Florida 32399-0750 Lynda L. Goodgame, Esquire General Counsel Northwood Centre 1940 North Monroe Street Tallahassee, Florida 32399 Angel Gonzalez Executive Director Board of Professional Engineers 1940 North Monroe Street Tallahassee, Florida 32399
The Issue The issues in this case are whether certain questions within the June 2002 construction building contractor examination are invalid, and whether Petitioner should receive credit for certain answers scored as incorrect.
Findings Of Fact In June 2002, Petitioner sat for the construction building contractor examination. Shortly following the exam, Petitioner was advised that he incorrectly answered 17 of the 50 exam questions and did not attain the minimum passing score of 70 percent, but received a failing scaled score of 66 percent. Petitioner timely challenged the validity and scoring of eight questions, including questions 8, 14, 17, 33, 34, 38, 43, and 44. In order for Petitioner to acquire a passing score, Petitioner must prove that certain challenged questions are invalid or demonstrate that he is entitled to receive credit for his answers. Specifically, Petitioner must demonstrate that either three questions should be stricken from the exam providing Petitioner with 70.2 percent, two questions should be stricken and one answer scored as correct providing Petitioner with 70.8 percent or two answers should scored as correct providing Petitioner with 70 percent. QUESTION 8 Exam Question 8 asks, "According to AIA-A201, who determines the interest rate that the contractor can charge on due and unpaid payments?" Petitioner's expert, Mr. Uman, argues that the parties to the contract are not defined within the question and it is therefore misleading. However, the credited answer D, "all the parties must agree on the rate" is within the provided reference material and is clearly the best answer. It is not misleading and Petitioner's argument lacks merit. In addition, 89.47 percent of the test-takers correctly answered Question 8. QUESTION 14 Exam Question 14 is wordy and involves computations. It requires the test-taker to calculate the number of "labor" hours required per 100 pieces to build a wall, given certain pricing and wall construction information. Question 14 is ambiguous and confusing on its face. While the question asks for labor hours, the facts provide a fixed combined hourly cost for a mason and laborer's hour. There is no distinction made between "labor" hours and a "laborer's" hours. Mr. Collier admitted that there is some apparent confusion between "labor" costs and the "laborer's" costs. Mr. Palm further agreed and indicated that he fully understood Petitioner's rationale to divide the labor costs in half and choose answer A. Furthermore, it is clear that Petitioner's perception of the question was not unique. In fact, only 46.62 percent of the test-takers correctly answered Question 14. QUESTION 17 Exam Question 17 asks, "During the bid process, which document has priority in the event of conflicting information?" Clearly, the correct answer is B, "addenda." Petitioner's argument regarding "competitively bid projects" is without merit. Mr. Palm succinctly explained that Petitioner's selection was obviously incorrect because "plans don't change during the bid process unless there is an addenda issued." Moreover, 75.56 percent of the test-takers correctly answered Question 17. QUESTION 33 Exam Question 33 identifies a situation that where drawings differ from written specifications and where there is no legal precedent that one is more binding than the other. The question specifically calls for the best procedure according to the listed and available reference. While Mr. Uman argues that the answer does not appear within the reference material in a clear manner, the exact text of the question and answer are in fact within the material. Petitioner's argument lacks credibility. QUESTION 34 Exam Question 34 asks the test-taker "what is the EARLIEST workday for completing the masonry work?" given the number of crew, the number of hours required, and the ratio constant of the crew. Although 80.45 percent of the test-takers correctly answered Question 34, Mr. Uman argues that the question could have been answered without reference to the Builder's Guide to Accounting material and therefore, was misleading. Petitioner's argument is devoid of common sense. QUESTION 38 Exam Question 38 asks the test-taker to identify the activity that "a specialty structural contractor is qualified" to perform. Petitioner's expert, Mr. Uman, again argues that the question is misleading since the credited correct answer "perform non-structural work" is not written verbatim in the provided reference material. To the contrary however, all of the alternative choices are clearly listed in the reference material as activities specifically prohibited by specialty structure contractors. Furthermore, page 2B17 to 61G415.015 of the Contractor's Manual specifically states that: The specialty structure contractor whose services are limited shall not perform any work that alters the structural integrity of the building including but not limited to roof trusses. Respondent's experts, Mr. Collier and Mr. Palm, agree that Question 38 is clear. Moreover, 53.38 percent of test- takers correctly answered the question. While the question appears to require enhanced reasoning skills and is generally more difficult, it is not misleading. Petitioner's assertions are without merit. QUESTION 43 Exam Question 43 asks, "Which accounting method should be used by a contractor if the contractor is unable to reasonably estimate the amount of progress to date on a job or the total costs remaining to fulfill the contract?" Mr. Uman argues that the question is ambiguous and the reference material is "not terribly clear." He further alleges that when a contractor cannot estimate progress, the contractor cannot establish a "completed contract method," the credited correct answer. Respondent's experts disagree. While it is true that Mr. Palm agreed that all of the choices are accounting methods which is inconsistent with Mr. Collier's testimony, the reference material is clear. In fact, 58.65 percent of the test-takers correctly answered Question 43. Petitioner presented insufficient evidence that he should receive credit for his answer or that Question 43 should be invalidated. QUESTION 44 Exam Question 44 provides detailed information regarding a standard form contract and asks, "Based ONLY on the information given above, what is the amount of the current payment due?" In addition, however, as Mr. Uman points out, the standard form referred to in the problem was mistakenly misidentified as Form 201 instead of Form 702. While it is clear that the referenced form was mislabeled, the precise form number was incidental, unrelated to the question, and unnecessary to compute the answer. In fact, Mr. Palm explains that the problem was "just a mathematical exercise." According to Mr. Collier, the question was not misleading, and the incorrect reference was irrelevant. "It's simple math, industry knowledge." Furthermore, Petitioner's answer is clearly incorrect because "he failed to deduct the retainage." Finally, 54.89 percent of the test-takers correctly answered Question 44.
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is RECOMMENDED that a final order be entered invalidating only Question 14, re-computing Petitioner's examination score, and dismissing his challenge. DONE AND ENTERED this 1st day of October, 2003, in Tallahassee, Leon County, Florida. S WILLIAM R. PFEIFFER Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 SUNCOM 278-9675 Fax Filing (850) 921-6847 www.doah.state.fl.us Filed with the Clerk of the Division of Administrative Hearings this 1st day of October, 2003. COPIES FURNISHED: Nickolas Ekonomides, Esquire 791 Bayway Boulevard Clearwater, Florida 33767 Charles F. Tunnicliff, Esquire Department of Business and Professional Regulation 1940 North Monroe Street, Suite 60 Tallahassee, Florida 32399-2202 Nancy P. Campiglia, General Counsel Department of Business and Professional Regulation Northwood Centre 1940 North Monroe Street Tallahassee, Florida 32399-2202 Robert Crabill, Executive Director Construction Industry Licensing Board Department of Business and Professional Regulation Northwood Centre 1940 North Monroe Street Tallahassee, Florida 32399-0792
The Issue Whether the petitioner is entitled to credit for the answers given to the challenged questions in the General Contractor’s examination administered October 18, 1995.
Findings Of Fact Based on the oral and documentary evidence presented at the final hearing and on the entire record of this proceeding, the following findings of fact are made: The Department of Business and Professional Regulation, Construction Industry Licensing Board, is the state agency with jurisdiction over the examination and regulation of general contractors in the State of Florida. Sections 489.107(4) and .113(1), Florida Statutes. Mr. Simmons sat for the General Contractor examination on October 18, 1995, and received a failing grade of 68.75 percent on the business and financial administration portion of the examination. Even though he passed the other two portions of the examination, Mr. Simmons failed the examination as a result of the failing grade on this portion of the examination. Mr. Simmons subsequently filed a timely challenge to unspecified test questions on the business and financial administration portion of the examination. He presented evidence at the hearing concerning the sufficiency of his answers to questions 13 and 22 of the financial administration section of the examination and claimed that he would have presented evidence relating to questions 18 and 39 of the business administration section but could not because he was not certain that the questions included in the review materials provided to him by the respondent were the same as the questions included in the test booklet he used on October 18, 1995. Question 13 of the financial administration section of the examination is an objective, multiple choice question. The applicant is to choose the correct answer from among four choices. The correct answer to question 13 is “C," but Mr. Simmons incorrectly chose “B.” Question 22 of the financial administration section of the examination is an objective, multiple choice question. The applicant is required to choose the correct answer to the question from among four answers provided and is to choose the correct answer based only on the information included in the question. The correct answer to question 22 is “C," but Mr. Simmons incorrectly chose “A.” Mr. Simmons failed to prove that questions 18 and 39 of the business administration section of the examination included in the review manual provided to him by the respondent were not the same questions included in the test booklet he used on October 18, 1995. Because he failed to present any evidence regarding the sufficiency of his answers to these questions, he is deemed to have abandoned any substantive challenge to them. Question 13 of the business administration section of the examination is clear and unambiguous, and only one correct answer was included among the answer choices. The correct answer is found in the reference material which Mr. Simmons was permitted to use while he was taking the examination. The respondent correctly gave no credit to Mr. Simmons for his answer to this question because it was the wrong answer. Question 22 of the business administration section of the examination is clear and unambiguous, and only one correct answer was included among the answer choices. The method for determining the correct answer from the information provided in the question is contained in the reference materials Mr. Simmons was permitted to use while he was taking the examination. The respondent correctly gave no credit to Mr. Simmons for his answer to this question because it was the wrong answer.
Recommendation Based on the foregoing findings of fact and conclusions of law, it is RECOMMENDED that the Department of Business and Professional Regulation, Construction Industry Licensing Board, issue a final order dismissing Alfred Simmons’s challenge to the subject examination and that the examination questions and answers provided at the hearing be sealed and not open to public inspection. DONE AND ENTERED this 3rd day of January, 1997, in Tallahassee, Leon County, Florida. PATRICIA HART MALONO Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (904) 488-9675 SUNCOM 278-9675 Fax Filing (904) 921-6847 Filed with the Clerk of the Division of Administrative Hearings this 3rd day of January, 1997. COPIES FURNISHED: Alfred Simmons 7755 West Kismet Street Miramar, Florida 33023 R. Beth Atchison, Assistant General Counsel Department of Business and Professional Regulation Construction Industry Licensing Board Northwood Centre 1940 North Monroe Street Tallahassee, Florida 32399-0750 Lynda L. Goodgame General Counsel Department of Business and Professional Regulation Northwood Centre 1940 North Monroe Street Tallahassee, Florida 32399-0792 Richard Hickok, Executive Director Construction Industry Licensing Board 7960 Arlington Expressway, Suite 300 Jacksonville, Florida 32211-7467
The Issue Whether Petitioner is entitled to additional credit for his responses to question numbers 21 and 24 of the Principles and Practice of Engineering Examination administered in April 1998.
Findings Of Fact Petitioner took the April 24, 1998 professional engineering licensing examination with an emphasis in civil engineering. A score of 70 is required to pass the test. Petitioner obtained a score of 69. In order to achieve a score of 70, Petitioner needs a raw score of 48. Therefore, Petitioner is in need of at least one additional raw score point. Petitioner is challenging question numbers 21 and 24. They are both multiple-choice questions and worth one point each. Exhibit 10 contains a diagram for the candidate's use in answering question numbers 21 and 24. Question 21 requires the examinee to calculate the percentage of wooded land on the diagram. The diagram contains a rectangle labeled "woodlot," and within the rectangle are three non-contiguous areas marked with schematics of trees. The Petitioner reduced the percentage of wooded area to conform to the portion of the area labeled "woodlot" marked with schematics of trees. In regard to question number 21, the Petitioner asserts that as a matter of convention, by failing to put the trees everywhere in the wooded lot, one may assume that there are trees only where there is a schematic of the trees. The Petitioner's challenge was rejected on the basis that the scorer opined that it is standard practice that drawings are only partially filled with details, and the most reasonable interpretation of the site plan drawings is that the woodlot fills the entire area enclosed by the rectangle. John Howath, a professional engineer, testified regarding accepted conventions in engineering drawings. In Howath's opinion the drawing on the examination used inconsistent methodologies and was confusing regarding whether all of the area designated by the label or "call out" of woodlot was in fact wooded. Both the Petitioner and Mr. Howath referred to drawings in the Civil Engineering Reference Manual which showed areas on drawings totally covered with visual indications of a particular material or condition. Peter Sushinsky, a professional engineer, testified as an expert for the Respondent. Mr. Sushinsky acknowledged the Petitioner's exhibits; however, Mr. Sushinsky noted that these were only a few examples of drawings that are available. Mr. Sushinsky referenced construction drawings he had seen in his practice with partial "cross-hatching" just like the diagram on the examination. In sum, Mr. Sushinsky's experience was that diagram might be totally or partially "cross-hatched." In Mr. Sushinsky's opinion it was not a bad diagram, only subject to a different interpretation by a minor group. Question number 24 asked the candidate to calculate the weir peak discharge from the catchment area using the rational formula. The Petitioner asserts the question is misleading and should read, "What is the peak discharge from the watershed?" The Petitioner bases his assertion on the ground that the "rational formula" is used to compute discharge from a watershed not a weir, as mandated by the question. The scorer did not address the Petitioner's concerns. The scorer stated, "It is clear from the item statement that the weir equation is not to be used." However, the questions ask the candidate to compute the weir discharge. Jennifer Jacobs, a professor of engineering, testified regarding the rationale formula that it was used to calculate watershed discharge and not weir discharge. All experts agreed that the rational formula is not used to compute weir discharge. The experts all agree that the question was confusing because the rational formula is not used to calculate the discharge from a weir. The Respondent's expert justifies the answer deemed correct on the basis that if one uses the rational formula and computes the watershed discharge, one of the answers provided is close to the result. The Respondent's expert calculated the watershed discharge as 230.6 cubic feet per second (cfs). The answer deemed correct was 232 cfs. The expert stated the weir attenuates flow. If the weir attenuates flow one would expect an answer less than 230.6 cfs., not an answer equal to or greater than 230.6 cfs. The amount of attenuation is based upon the physical features of the impoundment area and the mouth of the weir. Weir Attenuation varies. The only answers smaller than 230.6 are 200 or 32. Is the 232 cfs. answer wrong because it does not allow for attenuation by the weir? How much did the weir attenuate the flow? Under these facts, the question is capricious. The Respondent argues that the Petitioner didn't follow instructions while acknowledging that the "correct" answer is not the answer to the question that was asked.
Recommendation Based upon the foregoing Findings of Fact and Conclusions of Law set forth herein, it is RECOMMENDED: That the Respondent enter a final order awarding Petitioner two raw points and a passing score on the Principles and Practice of Engineering Examination. DONE AND ENTERED this 20th day of May, 1999, in Tallahassee, Leon County, Florida. STEPHEN F. DEAN Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 SUNCOM 278-9675 Fax Filing (850) 921-6847 www.doah.state.fl.us Filed with the Clerk of the Division of Administrative Hearings this 20th day of May, 1999. COPIES FURNISHED: Mark W. Nelson 720 Northwest 31st Avenue Gainesville, Florida 32609 Natalie A. Lowe, Esquire Board of Professional Engineers 1208 Hays Street Tallahassee, Florida 32301 Dennis Barton, Executive Director Board of Professional Engineers 1208 Hays Street Tallahassee, Florida 32301 William Woodyard, General Counsel Department of Business and Professional Regulation 1940 North Monroe Street Tallahassee, Florida 32399-0792
Findings Of Fact Petitioner, William Everett Warriner, is an applicant for licensure by examination to practice architecture in the State of Florida. The architecture examination in the State of Florida consists of two parts, one of which is a written examination given in December of each year, and the other of which is a Site Planning and Design Test given in June of each year. Petitioner meets all requirements for admittance to the licensure examination. Petitioner took the Site Planning and Design Test portion of the National Council of Architectural Registration Boards (NCARB) in June, 1982. This portion of the examination is a twelve-hour sketch problem involving design and site consideration in which the applicant is graded on his or her design solution to the program requirements furnished, which are identical for each candidate at a given examination, on the basis or certain stated criteria, by trained graders who are registered architects. The examination is administered by the Department of Professional Regulation and is supplied to the State of Florida, as well as to all of the jurisdictions of the United States by NCARB. The examination involves the design of a structure, in this case a small municipal airport terminal building, by an applicant, including requirements for placing the structure on the site, elevations, facades, floor plans, and other aspects. The applicant is supplied with a preexamination booklet which sets forth the architectural program to be accomplished and the various requirements to which the applicant is expected to apply himself in order to receive a passing grade. At the time of the actual examination, the applicant is furnished other information to enable him to more adequately design the facility and demonstrate his command of the architectural requirements. In general, the examination was designed to require the applicant to design a solution to the site plan and the building design problems submitted to him by NCARB. The pertinent portion of the examination allows the examination graders to determine whether an applicant is able to coordinate the various structural design, technical aesthetic, energy, and legal requirements in order to resolve the design and site plan problem. The grading of the Site Planning and Design Test is accomplished by the review of the candidate's product by at least three architects selected by the various architectural registration boards of several states, who are given training by NCARB in an effort to, as much as is possible, standardize their conceptions of the minimal competence required for a passing grade. Each reviewer then assesses the product submitted by candidates/examinees on a "blind grading" basis, that is without knowledge of the identity or geographical origin of the submitter, or of the grade assigned the product by other reviewers/graders. The graders assess the product with a view toward identifying areas of strength or weakness within an overall determination of satisfaction and assign a holistic numerical score ranging from "0" (fail) to "4" (pass). Grades "1" (incomplete) and "2" (poor) are failing grades, and grades "3" (minimally acceptable) and "4" (good) are passing grades. If grades "1" or "2" are awarded, the graders are required to indicate the examinee's weakness, but these indications of weakness must not be arbitrary. All solutions are graded by three examiners, except those solutions to which a "0" has been given by one examiner. Solutions which are not clearly passing or failing are graded by a fourth examiner. In order for an applicant to pass, he must receive passing grades from at least two examiners, who independently grade his solution to the problem. A passing grade, as was stated above, is defined as a holistic grade of "3" or "4" as set forth in Rule 21B-14.04, Florida Administrative Code. Petitioner received a grade of "2", which is a failing grade, from each of the three graders who graded his examination. Though the Petitioner demonstrated an effort to comply with the criteria set forth in the examination and indicated in each area identified as weak on the examination grade report wherein he felt he had achieved the desired goal and standard, Mr. Burke, a registered architect and a member of the Florida Board of Architecture, identified several material areas wherein the Petitioner failed to observe program requirements. In the Site Planning and Site Design area, Petitioner has not shown any details as to how handicapped individuals would get across the median in the parking area in that there is no showing of a ramp from the pavement over the curb and across the median, nor is there any showing of lighting over the curbs for the handicapped. Further, on the issue of service area location, as drawn by Petitioner, this layout would require all service vehicles to pass in front of the airport through passenger traffic to get to the service cut for the service drive, which, in itself, is located too close to the baggage handling area. In addition, the site aesthetics were deficient in completeness and clarity in that landscaping was not shown, nor was appropriate consideration given to water flow and drainage. Additional deficiencies were demonstrated in the areas of building planning and design. The functional relationships of programmed areas were basically accomplished, but major problems exist in the general lobby area. While the test problem calls for the display area to be in the general lobby area, in Petitioner's solution, they are away from the ticket area and somewhat hidden. Those facilities which need exposure do not get it. Pedestrian circulation between the baggage claim area and the lobby exits is obstructed by the location of the car rental booths. In addition, traveling from the deplaning area on the second floor to the baggage claim area on the ground floor is made too difficult. The solution's conformity to barrier-free requirements is weak. There is little or insufficient protection from the elements at the entrances and exits. The solution's requirement for 74-foot trusses in the terminal creates excessive wasted volume in the attic area, and the overall form is awkward. In addition, one page of the problem is not completed, and emergency exits are not shown as required. In the section involving technical aspects of the plan, the first two sub-areas were marked weak primarily because of the incompleteness of the technical plan for the foundation. Further, only a very few technical notes appear on the solution, providing insufficient information, and the use of wood for a public building is dangerous. Petitioner disagrees with his grade and presented evidence to show that his failure was a marginal one. He feels he has rebutted approximately 75 percent of the failing items and passed the written part of the examination (Part B) on the first attempt. He has been involved in residential design for twelve years and has considerable experience. He feels the comment on the handicap ramps is valid, but that is only one part of the problem. There is sufficient other provision made for the handicapped in his solution. The lobby displays would have the least likelihood of being seen if placed in the ticket area as suggested. He opines that the covered walkways over the two major entrances are sufficient, and it would be superfluous to have covers over every door. Repeated rebuttal, such as those items listed, display Petitioner's difference of opinion with the grade assigned his problem, but not that his examination was graded in an arbitrary or capricious way or in a manner different than that utilized in grading the examination of every candidate taking the same examination throughout the United States. Also, his excuse for incompleteness that he only had twelve hours in which to complete the problem is invalid. The same time was allotted all candidates, including those who passed.
Recommendation Based upon the foregoing Findings of Fact and Conclusions of Law, it is, therefore, RECOMMENDED: That a final order be entered finding that Petitioner has failed to achieve a passing score on the June 1982 architecture examination and upholding the grade awarded to Petitioner on that examination. RECOMMENDED this 3rd day of January, 1983, in Tallahassee, Florida. ARNOLD H. POLLOCK, Hearing Officer Division of Administrative Hearings The Oakland Building 2009 Apalachee Parkway Tallahassee, Florida 32301 (904) 488-9675 Filed with the Clerk of the Division of Administrative Hearings this 3rd day of January, 1983. COPIES FURNISHED: John J. Rimes, III, Esquire Assistant Attorney General Department of Legal Affairs The Capitol Tallahassee, Florida 32301 Mr. William Everett Warriner 305 North East Fifth Avenue Gainesville, Florida 32601 Mr. Herbert Coons, Jr. Executive Director Board of Architecture Department of Professional Regulation 130 North Monroe Street Tallahassee, Florida 32301 Mr. Fred Roche Secretary Department of Professional Regulation 130 North Monroe Street Tallahassee, Florida 32301
