The Issue The issue for determination is whether Petitioner is eligible for licensure by the Board of Professional Engineers.
Findings Of Fact In October 1997, Jamal Jamilzadeh (Petitioner) took the Principles and Practice part of the Environmental Engineering Examination (Examination). The minimum score required to pass the Examination was The Department of Business and Professional Regulation, Board of Professional Engineers (Respondent) notified Petitioner that he had not successfully completed the Examination, having received a score of 68. The Examination is a national examination and is graded by national examiners, i. e., the National Council of Examiners for Engineering and Surveying (NCEES). A scoring plan is used for grading each question. Each person sitting for the Examination is referred to as a candidate. By letter dated April 8, 1998, Petitioner notified Respondent that he was challenging questions numbered 320 and 323 on the Examination. In his letter, Petitioner indicated the basis for the correctness of his answers to both questions. For question numbered 323, Petitioner included together with his letter a supporting statement, regarding the correctness of his answer, from a Certified Environmental Trainer (CET). At hearing, Petitioner withdrew his challenge to question numbered 320. Only question numbered 323 was now being challenged by Petitioner. For question numbered 323, the highest score achievable was 10. Petitioner received a score of 2. Petitioner's Examination was returned to the NCEES for review and rescoring. NCEES' rescorer used the same scoring plan that was used for the Examination. NCEES' rescorer had, for review and consideration, Petitioner's letter dated April 8, 1998, including the supporting statement of Petitioner's CET. NCEES' rescorer did not have access to and had no knowledge of Petitioner's original score for question numbered 323. NCEES' rescorer recommended that Petitioner receive no additional points for question numbered 323. Question numbered 323 contains three parts regarding an underground storage tank. The first part required a candidate to justify whether the underground storage tank was a confined space. NCEES' rescorer recommended that Petitioner receive full credit for this part. At hearing Petitioner's CET agreed that Petitioner should receive full credit for the first part. However, Respondent's expert disagreed and opined that Petitioner should not receive full credit because Petitioner provided only one of the three necessary requirements for a correct response. Petitioner correctly answered the first part.2 Petitioner should receive full credit for the first part. The second part required a candidate to list seven confined space program elements for initial entry and inspection of the underground storage tank. NCEES' rescorer found Petitioner's answer to be partially correct in that Petitioner failed to address three major procedural elements. At hearing, Petitioner's CET opined that Petitioner addressed five of the seven elements. However, Respondent's expert opined that, even though Petitioner addressed six elements, the elements addressed by Petitioner were different variations of two of the seven elements. Petitioner correctly addressed five of the seven elements.3 Petitioner should receive additional credit for part two. The third part required a candidate to specify the correct type of respiratory protection and to justify the answer. In reviewing this part, NCEES' rescorer also used the caveats or conditions specified in the supporting statement of Petitioner's CET, which was included with Petitioner's letter dated April 8, 1998. NCEES' rescorer found Petitioner's answer to be incorrect in that Petitioner failed to completely justify his answer. At hearing, Petitioner's CET opined that Petitioner's answer contained the concept, the intent, and the basis needed, which showed that Petitioner had the knowledge to answer the question presented, but that Petitioner's answer was not as detailed as it could or should have been. Respondent's expert opined that Petitioner failed to reference any monitoring with respect to the use of an air purifying respirator. Petitioner's CET agreed that monitoring was required but opined that monitoring was contained in Petitioner's answer. Petitioner failed to completely justify his answer in part three.4 Petitioner should receive no additional credit for part three. Petitioner's answers were not arbitrarily or capriciously graded. The grading process was not devoid of logic and reason, except for part two of question numbered 323.5 The scoring plan was properly used. The evidence presented was insufficient to warrant additional credit to Petitioner on question numbered 323. According to the scoring plan, Petitioner is entitled to 2 additional points, bringing his total score to 4 for question number 323. Petitioner's score on the Examination should be 70. Petitioner has obtained the minimum score required to pass the Examination.
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is RECOMMENDED that the Department of Business and Professional Regulation, Board of Professional Engineers enter a final order finding Jamal Jamilzadeh eligible for licensure. DONE AND ENTERED this 30th day of April, 1999, in Tallahassee, Leon County, Florida. ERROL H. POWELL Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 SUNCOM 278-9675 Fax Filing (850) 921-6847 www.doah.state.fl.us Filed with the Clerk of the Division of Administrative Hearings this 30th day of April, 1999.
The Issue The issues for determination are whether Respondent failed to maintain good moral character, in violation of Sections 943.1395(6) and (7), Florida Statutes (1995)1 and Florida Administrative Code Rules 11B-27.0011(4)(b) and (c),2 by making false statements to police officers; and, if so, what, if any, penalty should be imposed.
Findings Of Fact Petitioner is the governmental agency responsible for certifying and regulating law enforcement officers in the state. Respondent is certified as a law enforcement officer pursuant to certificate number 107546. On January 18, 1993, Respondent and Deputy Paolo Donisi were employed by the West Melbourne Police Department (the "Department"). The Department maintains intoxilyzers for use in prosecuting driving violations involving alcohol.3 Rule 10D-42.024(1)(b) requires the Department to assure that monthly preventative maintenance procedures are conducted on each intoxilyzer by a certified intoxilyzer operator no later than the 31st of each month. Preventive maintenance procedures involve 12 tests that require 1-1.5 hours to complete. Monthly maintenance procedures include a check of the LED display, the diagnostic system, and a chemical test. The chemical test includes a mouth alcohol test and four ethanol tests. The certified intoxilyzer operator maintains a record of the preventative maintenance procedures he or she performs ("maintenance records"). Respondent was a certified intoxilyzer operator for the Department from 1989 through December, 1992. After December, 1992, Respondent's certification as an intoxilyzer operator expired. Respondent performed intoxilyzer preventive maintenance procedures for the Department through May, 1992. In May, 1992, Respondent requested that he be relieved of his responsibility to perform the Department's monthly maintenance procedures. Respondent performed the monthly preventive maintenance procedures for the Department in June, 1992, but did not complete the maintenance records. Sgt. Charles Schrum, a certified intoxilyzer operator, performed the Department's preventative maintenance from July through November, 1992. After November 29, 1992, Sgt. Schrum performed the next preventative maintenance on January 4, 1993. Sgt. Schrum did not request Respondent to perform the preventative maintenance in June, 1992. Sgt. Schrum never expressed any reservations or concerns about his own ability to perform the monthly maintenance. Sgt. Schrum never asked Respondent to perform a backup or supplemental monthly maintenance test. Nor was Respondent authorized to perform such a test. On January 18, 1993, the State Attorney's Office requested that Deputy Donisi provide intoxilyzer maintenance records for the month of December, 1992. Deputy Donisi was not certified as an intoxilyzer operator. He could not access the records himself and requested that Respondent assist him in obtaining the records. The two men entered the room where the intoxilyzer was kept. They found that the Department had not performed the required monthly maintenance for December, 1992. Deputy Donisi asked Respondent what they should do. Respondent stated that he would perform the required maintenance and back-date the maintenance records. Respondent told Deputy Donisi that he may want to wait outside the room so that he did not become involved in the late testing. Deputy Donisi waited in the doorway of the room. Respondent completed the preventive maintenance form in less than 15 minutes. Respondent did not perform any of the preventative maintenance tests on the intoxilyzer. Respondent completed the preventive maintenance form by writing numbers on the form from a previously completed preventive maintenance form. Respondent handed the completed preventive maintenance form to Deputy Donisi. He instructed Deputy Donisi to give the form to the State Attorney's Office. The form falsely stated that Respondent had actually performed the monthly maintenance procedures and that the procedures had been performed at 2:30 p.m. on December 22, 1992. Deputy Donisi subsequently discussed with Sgt. Schrum how easy and quick it was to perform the monthly intoxilyzer maintenance. Deputy Donisi asked how he could become certified as an intoxilyzer operator. Sgt. Schrum inquired about the time it took Respondent to complete the preventive maintenance form and concluded that Respondent could not have performed the monthly maintenance test in 15 minutes or less. Deputy Donisi retrieved the maintenance records from the State Attorney's Office. Deputy Donisi recommended that the State Attorney dispose of those cases affected by the December maintenance records without going to trial. Sgt. Schrum and Deputy Donisi filed affidavits with the Department. The Department began an internal investigation on March 17, 1993. Internal investigators conducted two interviews with Respondent on April 21 and May 10, 1993. During the interviews, Respondent falsely stated that he had performed the monthly maintenance for December, 1992, and that he had conducted the tests on December 22, 1992. Each interview was recorded and transcribed. During the interviews, Respondent stated that he did not record his test in the official log because Sgt. Schrum had a week or so to complete the required maintenance. Respondent stated that he recorded his test results in his field notes. Respondent's field notes are inconsistent with the test results Respondent recorded in the monthly maintenance report he gave to Deputy Donisi. Respondent explained the inconsistency as a transposition error. Department policy requires law enforcement officers, such as Respondent, to keep a daily activity or duty log as part of their regular duties. Respondent's duty log for December 22, 1992, does not include any reference to performing monthly maintenance tests for the intoxilyzer. Records of the computer aided dispatch system used by the Department show that Respondent was at the station on December 22, 1992, from 11:18 a.m. until 2:55 p.m. From 2:55 p.m. until 3:59 p.m., Respondent was dispatched three times. Respondent was dispatched at 3:01 p.m. to an accident but was preempted after 28 seconds. He was dispatched again at 3:29 p.m. to another accident and was on the scene at 3:31 p.m. He was back in service at 3:32 p.m.
Recommendation Based upon the foregoing Findings of Fact and Conclusions of Law, it is RECOMMENDED that Petitioner enter a Final Order finding Respondent guilty of violating Sections 943.13(6) and (7) and Rule 11B-27.0011(4); and suspending Respondent's certificate for one year, including the period, if any, that Respondent has been unemployed by the Department prior to the date of this Recommended Order. RECOMMENDED this 17th day of January, 1997, in Tallahassee, Florida. DANIEL MANRY Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-1550 (904) 488-9675 SUNCOM 278-9675 Fax Filing (904) 921-6847 Filed with the Clerk of the Division of Administrative Hearings this 17th day of January, 1997.
The Issue Whether Petitioner is entitled to additional credit for his solutions to four problems on the Principles and Practice of Engineering portion of the engineering licensure examination administered on October 30, 1998, by the National Council of Examiners for Engineers and Surveyors.
Findings Of Fact Based upon the evidence adduced at hearing and the record as a whole, the following findings of fact are made: On October 30, 1998, as part of his effort to obtain a Florida engineering license, Petitioner sat for the Principles and Practice of Engineering Examination (Examination). This is a national examination developed and administered by the National Council of Examiners for Engineers and Surveyors (NCEES). Petitioner chose to be tested in civil engineering. Petitioner received a raw score of 45 on the Examination. For the civil engineering specialization, a raw score of 45 converts to a score of 67. To pass the Examination, a converted score of 70 is needed. Petitioner formally requested (in writing, by letter dated March 26, 1999) that his solutions to Problems 120, 125, and 222 on the Examination be rescored. Petitioner's written request was made to Natalie Lowe of the Board, who forwarded it to the NCEES. Appended to Petitioner's letter to Ms. Lowe were two pages of "scratch paper" on which Petitioner had written during his post-examination review on March 19, 1999. On the first page were written comments he had made regarding the scoring of Problems 120 and 125. On the second page were the following written comments he had made regarding the scoring of Problems 220 and 222: 220 a, b, & c 2 parts b & c correct. Min. mark I should get[:] At least 5 instead of 2 and maybe 7. There is an error. 222 ok The NCEES's rescoring of Petitioner's solutions to Problems 120, 125, and 222 resulted in his receiving a raw score of 43 (or a converted score of 65, 5 points less than he needed to pass the Examination). The Board received the NCEES's rescoring results on May 12, 1999. The Board subsequently referred the matter to the Division to conduct an administrative hearing. At the administrative hearing that was held pursuant to the Board's referral, Petitioner challenged the grading of his solutions to Problems 120, 125, and 220 of the Examination, and indicated that he had "no dispute concerning the grading of [his solution to Problem] 222," notwithstanding that he had requested, in his March 26, 1999, letter to Ms. Lowe, that his solution to Problem 222 be rescored. Petitioner explained that he had made this request as a result of inadvertence and that he had actually intended to seek rescoring of his solution to Problem 220, not Problem 222. Problems 120, 125, and 222 were worth ten raw points each. Problem 120 contained four subparts (or requirements). Petitioner initially received four raw points for his solution to Problem 120. Rescoring did not result in any change to this score. Petitioner solved two subparts of Problem 120 correctly (subparts (a) and (b)). The solutions to the other two subparts of Problem 120 (subparts (c) and (d)), however, were incorrect inasmuch as Petitioner had neglected, in making the lateral force calculations and drawing the diagrams required by these subparts, to include the force attributable to the movement of the groundwater referred to in the problem. Therefore, in accordance with the requirements and guidelines of the NCEES scoring plan for this problem, the highest raw score that he could have received for his solution to this problem was a four, which is the score he received. Problem 125 contained three subparts (or requirements). Petitioner initially received a raw score of two for his solution to Problem 125. Upon rescoring, no change was made this raw score. Petitioner correctly solved only one of the three subparts of Problem 125 (subpart (c)). In his solution to subpart (a) of Problem 125, Petitioner did not provide, as required by this subpart, the quantities of water, cement, and aggregate necessary for the project described in the problem. Petitioner's solution to subpart (b) did not describe one of the acceptable slump increasing methods that the candidates were required describe in their solution to this subpart. Accordingly, giving Petitioner a raw score of two for his solution to Problem 125 was consistent with the requirements and guidelines of the NCEES scoring plan for this problem. Petitioner received a raw score of two for his solution to Problem 220. He did not request, in his March 26, 1999, letter to Ms. Lowe, a rescoring of his solution to this problem, and, as a result, his solution was not rescored. At the administrative hearing, Petitioner testified on his own behalf regarding the scoring of this solution and, during his testimony, contended that the score he received was too low; however, neither a copy of the problem, nor a copy of the NCEES scoring plan for this problem, was offered into evidence. Accordingly, the record is insufficient to support a finding that the score Petitioner received for his solution to Problem 220 was undeservedly low in light of the NCEES scoring plan for this problem. Petitioner initially received a raw score of eight for his solution to Problem 220. Rescoring resulted in this score being reduced two points to a six. Petitioner did not present any evidence supporting the position (which he advances in his Proposed Recommended Order) that he should have received a higher score for his solution to this problem, and, consequently, Respondent's expert, in his testimony at hearing, did not address the matter. While there were exhibits offered (by Respondent) and received into evidence relating to the scoring of Petitioner's solution to Problem 222, it is not apparent from a review of these exhibits that such scoring deviated from the requirements of the NCEES scoring plan for this problem (which was received into evidence as part of Respondent's Exhibit 12).
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is RECOMMENDED that a final order be entered rejecting Petitioner's challenge to the failing score he received from the NCEES on the Principles and Practice of Engineering portion of the October 30, 1998, engineering licensure examination. DONE AND ENTERED this 20th day of December, 1999, in Tallahassee, Leon County, Florida. STUART M. LERNER Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 SUNCOM 278-9675 Fax Filing (850) 921-6847 www.doah.state.fl.us Filed with the Clerk of the Division of Administrative Hearings this 20th day of December, 1999.
The Issue The basic issue in this case is whether the Petitioner should be given a passing grade on the April, 1987, professional engineering examination. At the hearing the specific issues in dispute were narrowed to whether the Petitioner should be given a higher grade on each of three questions on the examination. At the hearing the Petitioner testified on his own behalf and presented the testimony of two other witnesses. He also offered several documentary exhibits into evidence. The Respondent offered the testimony of one witness and also offered several exhibits. Subsequent to the hearing a transcript of the hearing was filed with the Hearing Officer and the parties were given a reasonable time thereafter within which to file their proposed recommended orders. Both parties filed post-hearing submissions containing proposed findings of fact, conclusions of law, and recommendations. The parties' proposals have been given careful consideration in the preparation of this recommended order. All findings of fact proposed by all parties are addressed in the Appendix which is attached to and incorporated into this recommended order.
Findings Of Fact Based on the stipulations and admissions of the parties, on the exhibits received in evidence, and on the testimony of the witnesses at hearing, I make the following findings of fact. The April, 1987, professional engineering examination required an applicant to work four problems in the morning session and four problems in the afternoon session, for a total of eight problems. In order to pass the examination, the applicant had to achieve an average score of six points for all eight problems, or a raw score of forty-eight points. Mr. Schmidt's examination was given a total score of forty points, comprised of scores as follows: 10, 8, 5, 5, 4, 4, 3, and 1. Mr. Schmidt, therefore, needs eight additional raw points in order to receive a passing grade on the examination. An additional requirement is that in order to receive a passing grade on the examination, the applicant must score six points or more on at least five of the eight questions. Mr. Schmidt is challenging three questions on the exam, questions 114, 411 and 418. On question 114, Mr. Schmidt was given a score of four. On question 411, Mr. Schmidt was given a score of five. On question 418, Mr. Schmidt was given a score of three. At the commencement of the hearing, the Respondent stipulated that Mr. Schmidt's score of question 418 should be increased to five. The Item Specific Scoring Plan (ISSP) is a device utilized to standardize graders so that a person grading a specific problem for various different candidates would consistently apply the same score to the same type of deficiency throughout the scoring process. There was an individualized Item Specific Scoring Plan for each problem given on the subject examination. Each of the Item Specific Scoring Plans contains objective criteria for assigning from 0 to 10 points to a candidate's answer to each question. There is no evidence that the Item Specific Scoring Plans are defective or arbitrary and capricious. The percentage of successful candidates on the chemical engineering examination has been rather low on recent examinations. Approximately 15% passed the April, 1986, exam. Only 2.9% passed the October, 1986, exam, and 25% passed the April, 1987, exam. During that same period of time the success rate was generally (but not always) higher for candidates for licensure in other fields of engineering. The grade of four given to Mr. Schmidt's response to question number 114 is consistent with the individualized Item Specific Scoring Plan for that question. The grade of five given to Mr. Schmidt's response to question number 411 is consistent with the individualized Item Specific Scoring Plan for that question. The grade of three given to Mr. Schmidt's response to question number 418 is not consistent with the individualized Item Specific Scoring Plan for that question. The parties have stipulated that Mr. Schmidt's grade on question number 418 should be at least five. The evidence is insufficient to show that Mr. Schmidt is entitled to a higher grade than five on question number 418.
Recommendation Based on all of the foregoing, I recommend that the Board of Professional Engineers issue a final order to the following effect: Increasing Petitioner's score on question 418 from three to five, leaving Petitioner's other scores unchanged, and assigning to Petitioner a final grade of forty-two. DONE AND ORDERED this 11th day of March, 1988, at Tallahassee, Florida. MICHAEL M. PARRISH, Hearing Officer Division of Administrative Hearings The Oakland Building 2009 Apalachee Parkway Tallahassee, Florida 32399-1550 (904) 488-9675 Filed with the Clerk of the Division of Administrative Hearings this 11th day of March, 1988. APPENDIX TO RECOMMENDED ORDER, CASE NO. 87-4175 The following are my specific rulings on all of the proposed findings of fact submitted by the parties. Findings proposed by Petitioner: Petitioner's proposed findings regarding problem number 114 are essentially correct summaries of the testimony as far as they go. Nevertheless, most of them have been omitted as unnecessary subordinate details, particularly in view of the further testimony of Dr. O'Connell to the effect that he had no quarrel with the ISSP for this question and to the testimony of both Dr. O'Connell and Dr. Hanley to the effect that the grade given to Petitioner on this question is consistent with the ISSP. Petitioner's proposed findings regarding problem number 411 have for the most part been rejected as irrelevant on the basis of testimony by both Dr. O'Connell and Dr. Hanley to the effect that the ISSP required evidence of a trial and error solution and that such a solution is not shown in the Petitioner's answer. Petitioner's proposed findings regarding problem number 418 are essentially correct summaries of the testimony as far as they go. Nevertheless, most of them have been omitted as unnecessary subordinate details, in view of additional evidence to the effect that the Petitioner's boxed answer to this question was not a reasonable answer. With regard to the penultimate paragraph of the Petitioner's proposed findings, the first two sentences are essentially correct, but also irrelevant, because the burden of proof is on the Petitioner rather than on `the Respondent. With regard to the remainder of the penultimate paragraph, I have made findings regarding the success rate of chemical engineers, but find that evidence, standing along, insufficient to establish any impropriety in the examination. The final paragraph of the Petitioner's proposed findings is more in the nature of argument than proposed facts. It may well be that the Petitioner received less prehearing information from the Respondent than he was entitled to receive, but those are matters which should be raised before rather than after the hearing, and are matters which are waived if not timely asserted. Findings proposed by Respondent: Paragraph 1: Accepted. Paragraph 2: Accepted. Paragraph 3: Omitted as unnecessary subordinate details. Paragraph 4: Omitted as unnecessary subordinate details. Paragraph 5: The essence of this paragraph has been accepted, but most details have been omitted an unnecessary. Paragraph 6: Accepted. Paragraph 7: Accepted. Paragraph 8: Accepted in substance. Paragraph 9: Accepted. Paragraph 10: Accepted. Paragraph 11: Accepted. Paragraph 12: Omitted as unnecessary subordinate details. COPIES FURNISHED: Mr. Daniel B. Schmidt 2209 Northeast 15th Terrace Gainesville, Florida 32601 H. Reynolds Sampson, Esquire Assistant General Counsel Department of Professional Regulation 130 North Monroe Street Tallahassee, Florida 32399-0750 William O'Neil, Esquire General Counsel Department of Professional Regulation 130 North Monroe Street Tallahassee, Florida 32399-0750 Allen R. Smith, Executive Director Board of Professional Engineers 130 North Monroe Street Tallahassee, Florida 32399-0750
The Issue Should Petitioner be deemed to have passed the Principles and Practice Chemical Engineering Exam?
Findings Of Fact Petitioner sat for the Principles and Practice of Engineering Examination on April 1O, 1987. The examination consisted of an applicant completing eight problems. On each problem, an applicant could receive from 1 to 10 points. A score of 5 or less was considered "unqualified", a score of 6 or more was considered "qualified". In order to pass the examination, an applicant needed to receive a total raw score of 48 or more, out of a possible score of 80. Petitioner received a raw score of 43 on his examination. Petitioner is challenging the scores for problems 110, 111, and 414. On problem 110, Petitioner received a score of 1. On problems 111 and 414, Petitioner received scores of 4. An Item Specific Scoring Plan (ISSP) was prepared for each examination problem. The ISSP was to be used by the persons grading the exam to assure that all solutions were graded consistently. The ISSP set forth the criteria to be used in grading the solution to a problem and set forth guidelines to be used in determining the points to be given to the solution. In addition to the ISSP, a preferred solution was prepared for each problem. The scores of 4 given for Petitioner's solutions to problems 111 and 414 are consistent with the ISSPs for those problems. The score of 1 given to Petitioner's solution for problems 110 is not consistent with the ISSP and the preferred solution for that problem. In order to receive a score of 8, the ISSP requires that a solution contain the following: ALL CATEGORIES satisfied, errors attributable to misread tables or calculating devices. Errors would be corrected by routine checking. Results reasonable, though not correct. An operable system is presented, containing all the key elements, but calculations may involve minor errors in math or transcribing data. Answers obtained for requirement (b) are reasonable. May have assumed an excessively high, but not obviously unreasonable, exit water temperatures, c.g. 150F. Petitioner's solution to problem 110 presented an operable system containing all key elements. The errors committed by Petitioner were caused by misreading a table and by transcribing the wrong data. Petitioner's solution to problem 110 should have received a score of 8. Petitioner should have received a raw score of 50 on the exam.
Recommendation Based upon the foregoing Findings of Fact and Conclusions of Law, it is RECOMMENDED that Respondent issue a final order giving Petitioner a passing grade on the Principles and Practice Chemical Engineering Examination. DONE and ENTERED this 11th day of May, 1988, in Tallahassee, Florida. JOSE A. DIEZ-ARGUELLES Hearing Officer Division of Administrative Hearings The Oakland Building 2009 Apalachee Parkway Tallahassee, Florida 32399-1550 (904) 488-9675 Filed with the Clerk of the Division of Administrative Hearings this 11th day of May, 1988. APPENDIX TO RECOMMENDED ORDER, CASE NO. 87-4412 Petitioner filed a post-hearing submission consisting of a two page letter containing nine paragraphs. The second to fourth paragraphs are numbered 1., 2. and 3. The findings of fact contained in the letter are addressed below. Paragraph numbers in the Recommended Order are referred to as "RO ." Paragraph in Letter Ruling and Recommended Order Paragraph Number First Not a finding of fact. Second, Third and Fourth Rejected as not supported by the evidence for solution to problems 111 and 414. Accepted for solution to problem 110. RO6-10. Fifth Argument. Sixth First sentence accepted. RO11. Rest of paragraph is irrelevant. Seventh Rejected as argument and contrary to weight of the evidence. Eighth Argument. Ninth Not a finding of fact. COPIES FURNISHED: Edwin K. Middleswart 3805 Dunwoody Drive Pensacola, Florida 32503 John B. Carr, Esquire 320 West Cervantes Pensacola, Florida 32501 Allen R. Smith, Jr. Executive Director Department of Professional Regulation 130 North Monroe Street Tallahassee, Florida 32399-0750 William O'Neil General Counsel Department of Professional Regulation 130 North Monroe Street Tallahassee, Florida 32399-0750
The Issue The issue for consideration in this case is whether Petitioner should be granted extra credit for questions numbers 320, 321, 322, and 323, for which he gave allegedly incorrect answers, on the October 1996, Environmental Engineer Examination administered by the Department.
Findings Of Fact At all times pertinent to the issues herein, the Board of Professional Engineers was the state agency responsible for the professional testing and licensing of professional engineers, and the regulation of the engineering profession in Florida. Petitioner is a graduate engineer, specializing in environmental engineering, who took the Environmental Engineer Examination administered by the Bureau on October 25 and 26, 1996. By Examination Grade Report dated February 17, 1997, the Bureau notified Petitioner that he had achieved a score of 67.00 on the examination; that a minimum score of 70.00 is required for passing the examination; and, therefore, that Petitioner had failed the examination. Petitioner thereafter filed an appeal of the examination results, challenging the grading of questions numbers 320, 321, 322, and 323 of the examination in question. Question 320 tests the candidate’s ability to understand the characteristics of pumps both in series and parallel. The engineering principle involved is Bernouli’s Theory. The problem is in two parts, A and B. The first part asks which of two impellers are in the pump, based on a given set of data using Bernouli’s Principle. Petitioner answered Part A correctly. Part B repeats Part A, except that the candidate has to recognize the difference between series and parallel pumps, and Petitioner did not get the question correct. The National Council of Examiners for Engineering and Surveying (NCEES) published a scoring plan for each question on the examination. The maximum award a candidate can receive on this question is “10.” The NCEES’ scoring plan for this question provides a score of “4,” which Petitioner received, when the candidate gets one part of the question correct and one part incorrect. To earn a score of “6” for the question, the candidate must present a correct parallel pump analysis, and in this case, Petitioner doubled head pressure instead of flow. Question 321 also consists of two parts and deals with a sewer which is facing overload based on population projections. A relief sewer is proposed and the candidate must do two things. He must first analyze the flow of the existing sewer, and then determine what the invert of the new sewer line would be at the outer end of that sewer In this instance, Petitioner got the second part of the problem correct but not the first. Petitioner started off correctly, but then incorrectly used a piece of information that was given. The problem must be solved using Manning’s Equation, and then checked for scouring velocity. Petitioner used the minimum velocity in determining what the flow is and, according to Mr. Hutchinson, this is not the way to solve the problem. Hutchinson suggests that in solving the problem, the candidate first finds out how much flow will exist in the years ahead by knowing the population and the flow per capita. Then, using Manning’s Equation, the candidate calculates the flow the existing sewer can take. Subtracting the second from the first, the answer is the flow the new pipe will have to be designed for. In the examination question, all the required information is given except the diameter, which is determined through the use of Manning’s Equation. Once that is done, the candidate must check the new scouring velocity. This is done by calculating the velocity in the new sewer to be sure it is in excess of the number given in the problem statement. Here, Petitioner took the minimum scouring velocity and used that figure to calculate the size of the pipe. As a result, he arrived at the wrong answer of ten inches, when the correct answer was twenty-four inches. Petitioner was awarded a grade of “4” for his answer to problem number 321. According to NCEES’ Scoring plan, a “4” reflects the candidate got only one of the two parts correct. Question 322 deals with a hazardous waste incinerator. The first part of the problem calls for a determination of the amount of air needed to complete combustion if the additional air (excess air) is 100 percent. This means twice the air needed to perfectly combust the material. The candidate must first put down the chemical equation, all the constituents of which are given in the problem. Then, the candidate must balance the equation, and for 100 percent excess air, one multiplies the air input by a factor of two. The second part of the problem asks for the amount of water necessary to quench the gasses. Petitioner did not correctly balance the chemical equation called for in the first part even though he made an effort, and he was given some credit for trying. His answer to the second part was twice what it should have been. Since Petitioner did not do either part of the problem correctly, the award of “4” for his answer was, in Hutchinson’s opinion, generous. Question 323 involves a situation wherein a vehicle which gives off carbon monoxide is used inside a facility. Some of the workers have experienced dizziness. Readings are given for the carbon monoxide levels. The candidate is asked to calculate several factors. The first is what the eight-hour time weighted exposure is. There are certain limits involved. The second is how much ventilation air would be necessary to reduce the concentration to a lower stipulated level in one hour. The size and other specifics of the facility are given. The third part of the question is a non-mathematical essay question wherein the candidate is asked to define the disadvantages of having a combustion engine internal to a facility. The fourth part of the problem asks why mere dilution of the pollution is not the solution to the problem. Petitioner answered the first part of the problem correctly. He overstated the amount of air called for in part two of the problem by a magnitude of two. Petitioner answered the third part of the problem correctly, but in the fourth part, provided only one of the two reasons called for. He was awarded a score of “4” for his answer to this problem. The NCEES’ scoring plan indicates a score of “4” is appropriate when the candidate gets the first part correct; commits a logic error in the second part; and provides only two of three answers called for in the combined third and fourth parts. This is exactly what Petitioner did. In Mr. Hutchinson’s opinion, none of the problems in issue here were beyond the scope of knowledge that should be expected of a candidate for licensure. In addition, the questions as written are not ambiguous or unclear, and they give the candidate enough information to properly answer the questions. The examination is not a test of a candidate’s ability to do mathematical calculations. The examiners look at the ability to calculate as something which a high school student should be able to do. What is being tested is the candidate’s understanding of the engineering particulars and concepts. For example, in problem 320, the examiners are testing the candidate’s understanding of the difference between parallel flow and series flow for a pump. Under the scoring plan, that issue carries as much or more weight that the ability to solve the mathematics. Petitioner did not demonstrate the requisite understanding. The examination is structured so as to administer four questions in the morning session and four questions in the afternoon session. The examination is made up of questions which are submitted by members of that committee of the NCEES which drafts the examinations. The proposed questions are tested by committee members who solve each question in no more than twenty minutes. If the committee members judge the question to be appropriate and acceptable, it goes into a question bank and is subsequently reviewed several times before it is first incorporated in an examination two or more years later. In each question, the subject matter and the language of the question are reviewed to determine that there is no trick information involved; that all information necessary to correctly solve the question is incorporated; and that the scoring plan is valid. If any changes are made to a question during the evaluation time, two additional independent reviews are required. The examination is given nation-wide at the same time. At that time, the NCEES selects fifty to sixty tests at random, which are sent in for scoring. Of those, ten are selected and sent to a monitor to insure uniformity of scoring and appropriateness of the scoring plan. Once the examination is determined to be satisfactory, fifteen expert judges are called in to evaluate the fifty to sixty tests and to review them for demonstrated minimum competence by the candidates whose examinations are under scrutiny. At that point, a minimum numerical score is reached, and the remaining tests are graded. In his cross examination of the Respondent’s expert Mr. Hutchinson, regarding not only each of the examination problems in issue but also the methodology of the development and grading of the examination, Petitioner prefaced his questions by extensive, comprehensive statements of his position as to the matter at issue. Notwithstanding frequent and repeated reminders by the Administrative Law Judge that the matters being expressed were unsworn and not testimony, and therefore could not be considered as evidence, Petitioner persisted. The majority of his comments and arguments made in his Proposed Findings of Fact and Conclusions are based on that material and it is impossible for the undersigned to recommend Petitioner be granted the relief he proposes, based on the evidence admitted at hearing, as his testimony, when received, was not persuasive. Petitioner also submitted at hearing, as his Composite Exhibit 1, a series of documents which, for the most part, include personal information regarding his credentials, and copies of the pleadings, orders, and correspondence which make up the case file. Also included was a letter from Petitioner’s supervisor testifying to his hard work, industry, and professionalism; and a breakdown of the raw scores he achieved on the examination in question. None of this has a significant bearing on the merits of his challenge.
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is recommended that the Board of Professional Engineers enter a Final Order in this matter denying Petitioner additional credit for his answers to problems 320, 321, 322, and 323, on the October 1996 Environmental Engineer Examination. DONE AND ENTERED this 17th day of February, 1998, in Tallahassee, Leon County, Florida. _ ARNOLD H. POLLOCK Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 SUNCOM 278-9675 Fax Filing (850) 921-6947 Filed with the Clerk of the Division of Administrative Hearings this 17th day of February, 1998. COPIES FURNISHED: Scott D. Walker 14535 Bruce B. Downs Boulevard Number 918 Tampa, Florida 33613 R. Beth Atchison, Esquire Department of Business and Professional Regulation 1940 North Monroe Street Tallahassee, Florida 32399-0792 Lynda L. Goodgame General Counsel Department of Business and Professional Regulation 1940 North Monroe Street Tallahassee, Florida 32399-0792 Angel Gonzalez Executive Director Board of Professional Engineers 1940 North Monroe Street Tallahassee, Florida 32399-0792
The Issue The issue in this case is whether Florida Administrative Code Rule 61G15-21.004(2) is an invalid exercise of delegated legislative authority.
Findings Of Fact The Parties Petitioner is an applicant for licensure as a professional engineer ("P.E.")2 in Florida, and currently works in the discipline of environmental engineering in Florida. His practice focuses primarily on water-related areas within that discipline. Petitioner is not currently licensed as a P.E. Respondent is a board within the Department of Business and Professional Regulation ("Department"). It is the state agency created pursuant to section 471.007, Florida Statutes, and charged with licensing professional engineers in Florida. Respondent is vested with the authority to adopt rules to implement chapter 471, regarding the regulation of the practice of engineering in Florida, as defined in section 471.005(7). Respondent adopted the Challenged Rule at issue in this proceeding. Statutory and Rule Background The engineering profession in Florida is regulated pursuant to chapter 471. A person may become licensed as a P.E. in Florida by applying for licensure, fulfilling specified educational and experience requirements, and 2 For purposes of this Final Order, the terms "professional engineer" or "P.E." will be used to refer to persons who are licensed engineers under chapter 471, Florida Statutes. either being endorsed for licensure as provided in sections 471.015(3) and (5), or passing the required licensure examinations. § 471.015, Fla. Stat. Pursuant to section 471.015, Respondent has adopted Florida Administrative Code Chapter 61G15-20, which codifies, in rule, the requirements for licensure as a P.E. in Florida. An applicant for licensure must be a graduate of a Board-approved engineering program; have the requisite number of years of engineering experience; and have passed the specified licensure exams. Fla. Admin. Code R. 61G15-20.0010. Section 455.217(1)(d) authorizes Respondent to adopt, by rule, the use of a national professional licensing examination that the Department has certified as meeting requirements of national examinations and generally accepted testing standards. To implement section 455.217(1), Respondent has adopted rule 61G15- 21.001, titled "Examination Designated; General Requirements." This rule requires that, unless an applicant qualifies for licensure by endorsement, he or she must pass the National Council of Examiners for Engineering and Surveyors ("NCEES") licensure exam. Part I of the NCEES exam is the Fundamentals of Engineering ("FE") exam, and part II of the NCEES exam is the Principles and Practice Exam ("PP") exam. Respondent has entered into a contract with NCEES to provide the FE and PP exams in Florida. A person must pass both the FE and PP exams to be licensed as a P.E. in Florida. § 471.015(1), Fla. Stat. The Challenged Rule states: "[t]he passing grade for Principles and Practice Exam is determined by the National Council of Examiners for Engineering and Surveying, where psychometric statistical methods are used to determine the level of performance that corresponds with minimal competence in the discipline." Fla. Admin. Code R. 61G15-21.004(2). The Challenged Rule is a subsection of rule 61G15-21.004, which is titled "Passing Grade." The Challenged Rule specifically and exclusively addresses the method for determining the passing grade on the PP exam. Sections 455.217(1)(d) and 471.013 are cited as the rulemaking authority for the Challenged Rule, and sections 455.217(1)(d) and 471.015(1) are cited as the law implemented by the Challenged Rule. The term "engineering," as used in section 471.005(7), includes the term "professional engineering," and defines the types of services and creative work that constitutes "engineering." An "engineer," as defined in section 471.005(5), includes the terms "professional engineer" and "licensed engineer," and means a person who is licensed to engage in the practice of engineering under chapter 471. By contrast, an "engineer intern," as defined in section 471.005(6), means a person who has graduated from a Board-approved engineering curriculum and has passed the FE exam. By definition, these are distinct terms. The term "engineer" is used to describe a person licensed as a P.E. under chapter 471, while the term "engineer intern" is used to described a person who may engage in the kinds of activities described within the term "engineering," as defined in section 471.005(7), but who is not licensed as a P.E. in Florida, and, therefore, is not authorized to hold himself or herself out as a licensed engineer in Florida. The Rule Challenge Petition The Rule Challenge Petition alleges four grounds under section 120.52(8) for invalidating the Challenged Rule. Alleged Invalidity of Challenged Rule under Section 120.52(8)(c) In paragraph 15 of the Rule Challenge Petition, Petitioner asserts that section 455.217(1)(c)—which, at the time the Rule Challenge Petition was filed, was cited as the specific authority for, and law implemented by, the Challenged Rule—did not authorize the Challenged Rule, so that the Challenged Rule enlarged, modified, or contravened the specific provisions of law implemented, pursuant to section 120.52(8)(c). As a result of Petitioner having filed the Rule Challenge Petition, Respondent discovered that it had not updated its citation of the specific authority for, and law implemented by, the Challenged Rule, when section 455.217(1) was amended and renumbered in 1997, so that section 455.217(1)(c) no longer was the correct citation to the law implemented by the Challenged Rule. Respondent requested the Department of State, Administrative Code and Register Section ("DOS"), to make a technical, non-substantive change to the Challenged Rule. As authorized by Florida Administrative Code Rule 1-1.010(10), DOS updated the statutory citation to section 471.217(1)(d), which is the correct citation to the law implemented by the Challenged Rule. This technical change nullifies the alleged invalidity ground set forth in paragraph 15 of the Rule Challenge Petition, and Petitioner concedes this. In paragraph 16 of the Rule Challenge Petition, Petitioner also alleges that the Challenged Rule enlarges, modifies, or contravenes section 455.217(1)(a), because the PP examination does not adequately and reliably measure an applicant's ability to practice the profession regulated by the Department. However, as discussed below, section 455.217(1)(a) is not cited as a specific provision of law implemented by the Challenged Rule, so cannot form the basis of a challenge to the Rule under section 120.52(8)(c). Alleged Invalidity of Challenged Rule under Section 120.52(8)(d) In paragraph 17 of the Rule Challenge Petition, Petitioner alleges that the Challenged Rule is invalid under section 120.52(8)(d) because it is vague, fails to establish adequate standards for agency decisions, or vests unbridled discretion in the agency. In support of this alleged invalidity ground, Petitioner asserts that the Challenged Rule is vague because "the level of performance on the PP exam is stated to correspond with minimal competency, yet there are no rules which provide definitive guidance to NCEES on what constitutes the general areas of competency in regards to engineering practice."3 Petitioner also alleges, in paragraph 19 of the Rule Challenge Petition, that the Challenged Rule is invalid under section 120.52(8)(d) because it equates passage of the PP exam with a 30-year practice experience requirement for licensure by endorsement set forth in section 471.015(5)(b). To this point, Petitioner states: "I can't think of anything more arbitrary than the principles and practice exam equating to near[-]retirement level experience."4 Alleged Invalidity of Challenged Rule under Section 120.52(8)(e) In paragraph 18 of the Rule Challenge Petition, Petitioner alleges that the Challenged Rule is invalid pursuant to section 120.52(8)(e) because it is arbitrary, for several reasons. Paraphrased, these reasons are: passage of the PP exam does not accurately reflect, or equate to, minimal competence in the discipline; the PP Exam does not accurately evaluate an individual's engineering ability level, but instead evaluates an individual's exam performance compared to average group exam performance; the PP exam does not reliably distinguish between minimal competence and incompetence to practice engineering, as evidenced by the fact that engineers who fail the PP exam still competently perform, and, thus, keep, their engineering jobs; passing the PP exam, by itself, does not certify an individual to competently perform any engineering service or creative work as defined in 3 Section 455.217(1)(b) requires, for each exam developed by the Department or a contracted vendor, that the general areas of competency covered by the exam be specified by rule. The last sentence of that subsection states that the requirements of subsection (b) do not apply to national exams, such as the NCEES PP exam, which are approved and administered pursuant to section 455.217(1)(d). Thus, the law implemented by the Challenged Rule does not require areas of competency to be specified in the Challenged Rule. 4 Because paragraph 19 of the Rule Challenge Petition alleges that the rule is arbitrary, the undersigned considers this paragraph to constitute a challenge to the Challenged Rule under section 120.52(8)(e), rather than under section 120.52(8)(d), as cited in the Rule Challenge Petition, and has addressed this ground in the Conclusions of Law section dealing with that alleged invalidity ground. section 471.005(7); the PP exam does not reliably determine if an examinee is minimally competent, due to an incorrect reference point; and the PP exam does not reliably distinguish between individuals whose practice of engineering would protect the public health and safety and those whose practice of engineering would constitute a danger to public health and safety. Alleged Invalidity of Challenged Rule under Section 120.52(8)(f) In paragraph 20 of the Rule Challenge Petition, Petitioner alleges that the Challenged Rule is invalid pursuant to section 120.52(8)(f) because it imposes regulatory costs on the regulated person, county, or city, which could be reduced by the adoption of less costly alternatives that substantially accomplish the statutory objectives. To this point, Petitioner proposes a state- conducted investigation of an applicant as a substitute for the PP exam, and contends that "there's a premium associated with a national exam. It's also clear that the regulatory cost imposed on me and all future examinees could be substantially reduced if the Board conducted their exam as an investigation and did not incorporate the NCEES exam." However, as more fully discussed below, this challenge ground is time- barred by section 120.541(1)(g), and, therefore, is not a legally-cognizable basis for invalidating the Challenged Rule in this proceeding. The Parties' Stipulated Facts Petitioner is an Engineer Extern, Texas EIT 56990. Pursuant to section 471.015(1), the Florida Engineers Management Corporation "shall issue a license to any applicant who the Board certifies is qualified to practice engineering and who has passed the [FE] exam and the [PP] exam." Both the FE exam and the PP exam are created by NCEES, pursuant to section 455.217(1)(d), which states, in pertinent part: "a board . . . may approve by rule the use of any national examination which the [Department of Business and Professional Regulation] has certified as meeting the requirements of national examinations and generally accepted testing standards pursuant to department rules." Pursuant to section 471.015(1), Petitioner passed the FE exam on September 3, 2016. Passing the FE is a prerequisite to take the PP exam. Petitioner registered to take the NCEES computer-based PP exam for environmental engineering on April 22, 2020. He paid the $350 exam fee and additional monies for test preparation material, and spent at least 100 hours preparing for the exam. He was notified, on April 6, 2020, by Pearson Vue, the test center company, that the exam was cancelled due to the Covid-19 pandemic. On April 7, 2020, Petitioner registered to take the exam on July 15, 2020, which was the earliest available date for taking the exam in his local area. On April 9, 2020, Petitioner canceled his registration for the July 15, 2020, exam and decided to apply for licensure as a P.E. without passing the PP exam. Pursuant to section 471.015(2)(a)1., on May 18, 2020, Petitioner filed his application for licensure with the Board. The application provides information stating that Petitioner meets the requirements of section 471.013(1)(a)1., and has at least four years of active engineering experience of a character indicating competence to be in responsible charge. In compliance with the education requirements of section 471.013(1)(a)1., Petitioner earned a Bachelor of Science degree in Chemical Engineering from the University of Florida, which is accredited by ABET. Petitioner provided an official transcript to the Board for verification purposes. In compliance with rule 61G15-20.002, Petitioner states in his application that he has at least four years of acceptable engineering practice, consisting of one year of experience equivalent through his Master of Engineering degree from the University of Texas at Austin, an ABET- accredited institution; and over three years of professional engineering work verified by licensed engineers. According to rule 61G15-20.002, an applicant must list three current personal references who are professional engineers. Thirteen licensed engineers submitted documentation to the Board regarding Petitioner's type of qualifying experience, level of engineering competency, and professional integrity. All references circled "yes" to the question "would you employ the applicant in a position of trust?" Two additional references in the engineering industry also provided their opinion on Petitioner's integrity and competence. On June 1, 2020, Petitioner received notification from the Board that his Florida 0901 1031-P.E. Endorsement application was incomplete. Petitioner addressed all items listed in a timely fashion to participate in the June 10, 2020, Board meeting. The Board stated during an informal hearing on June 10, 2020, that Petitioner would not be granted P.E. licensure due to not having passed the PP Exam. Petitioner is substantially affected by the Challenged Rule because it disqualifies him from becoming a licensed engineer in Florida without passing the PP exam. Findings of Fact Based on Evidence Presented at the Final Hearing A. Findings Regarding the Evidence Presented in Petitioner's Case Testimony of Witnesses Hoot and Grace David Hoot and Nigel Grace, both of whom are licensed professional engineers, testified regarding Petitioner's abilities and skills as a practicing engineer. Neither Hoot nor Grace was qualified as an expert in the field of psychometrics or related topics. Therefore, any opinions regarding these subject matters to which they testified at the hearing have been treated as personal opinions, rather than expert opinions. Hoot characterized Petitioner as a good, diligent young engineer, and described Petitioner's role in various projects on which they worked together. Based on his work with Petitioner, it was Hoot's personal opinion that Petitioner possesses the integrity and competence to work as a licensed engineer who would serve the public health and safety. Hoot testified that when he took the PP exam approximately 38 years ago, it was a free response exam. He stated that he was "not exactly sure" that a multiple-choice exam captures an examinee's ability to apply reason and judgment, but he acknowledged that he does not know how the PP exam currently is developed. He offered his personal opinion that it was possible for a competent engineer to fail the PP exam. Hoot also offered his personal view that engineering licensure constitutes a standard of care; however, he did not think anything provides a guarantee of competence. It was his personal view that experience tended to make one more competent. He also offered his personal view that as an engineer gains more experience and becomes more specialized, it is understandable that he or she would not score as well as an engineer who is gearing up toward the four-year experience goal which enables them to apply for licensure as a P.E. As Hoot put it, "life happens. You have children involved. You move jobs. I think you . . . maybe have less time to study. . . you get farther away from the study habits of . . . learning to be able to take tests." Grace, who is employed as a P.E. with Brown and Caldwell ("B&C"), a large U.S. engineering firm, testified regarding Petitioner's work experience while he was employed by B&C. Petitioner's experience included working on drinking water projects, utilities, upgrading process equipment, site evaluations, bench scale testing, and other projects. Based on Grace's work with Petitioner, it was his personal opinion that Petitioner possesses the integrity and competence to work as a licensed engineer who would serve the public health and safety. Grace took the PP exam approximately 28 years ago, and at the time, a major portion of the exam consisted of long-form written exam questions that provided the opportunity for examinees to provide free response answers and earn partial credit for partially-correct answers. He testified that the exam also had a multiple-choice component. Grace testified that, "based on instinct," he knows well-designed multiple-choice questions can provide the same insight into an examinee's decision-making judgment as long-form questions. Grace's personal view is that passing the PP exam does not guarantee competence, and it is possible for an engineer to be competent in some engineering disciplines but not others. Grace also agreed that it was logical that an examinee with approximately four years' experience would perform better on the PP exam because he or she would have better-honed test-taking skills and be fresher in some areas tested on the exam. By contrast, engineers who have practiced longer have more experience, but often have become specialized and further removed from the test-taking environment. Thus, passing a broad-spectrum exam becomes a bigger hurdle for engineers who have practiced longer. Petitioner's Testimony Petitioner testified on his own behalf at the final hearing. As discussed above, Petitioner holds a bachelor of science degree in chemical engineering from an ABET-accredited institution, and holds a master of engineering degree from an ABET-accredited institution. Although Petitioner, through his training as an engineer, is skilled at mathematics and statistics, he is not trained, and does not have any substantial experience, in the field of psychometrics.5 5 As discussed in greater detail below, psychometrics is a specialized field of study that concerns the theory and technique of objective psychological measurement of skills, knowledge, abilities, and educational achievement. Petitioner acknowledged that he does not have a degree in psychometrics and that he is not trained in, or knowledgeable about, preparing and administering high-stakes professional examinations. At the time he filed the Rule Challenge Petition, Petitioner had no personal knowledge of the work done by psychometricians; did not know what a "cut score" was; and was not familiar with the Modified Angoff Method, item response theory, the specifics of converting raw scores to scaled scores, or other any psychometric tools and methods employed to prepare and score the PP exam. Petitioner acquired some rudimentary knowledge in a few of these areas in preparing for the final hearing in this proceeding.6 Section 90.701(2), Florida Statutes, prohibits a lay witness from testifying as to an opinion regarding a matter involving specialized knowledge.7 Accordingly, it is determined that Petitioner is not competent to provide an expert opinion regarding psychometrics and related areas, such as item response theory. He was not qualified, tendered, or accepted at the final hearing as an expert in psychometrics or any related areas. Because Petitioner was not qualified to testify as an expert witness at the final hearing, Petitioner's testimony regarding psychometrics, item response theory, scaled scores, the Modified Angoff Method, high-stakes professional testing, and all other specialized subject matters, consisted of opinion testimony by a lay witness. The only instances in which a lay witness 6 Petitioner does not have any special knowledge, formal training, education, or experience in the specialized field of psychometrics. His knowledge about these areas was acquired by reading and study in preparation for the final hearing. Because Petitioner lacks special knowledge, experience, training, and education in psychometrics, he is not qualified to testify as an expert in psychometrics or related topics, such as item response theory. See Chavez v. State, 12 So. 3d 199, 205 (Fla 2009)(in determining whether a witness is qualified to render an opinion as an expert in a specialized field, the court must determine whether the witness is adequately qualified to render an opinion based on special knowledge, experience, training, or education). 7 Chesser v. State, 30 So. 3d 625 (Fla. 1st DCA 2010)(it is error for a court to accept opinion testimony of a lay witness in a specialized subject matter area). may provide opinion testimony are when the lay witness's opinion is based on firsthand knowledge through personal perception.8 As the undersigned explained during the final hearing, rather than excluding Petitioner's lay opinion testimony, she would consider these pertinent evidentiary principles in determining the weight to be afforded Petitioner's lay opinion testimony in this proceeding. Applying these evidentiary principles, it is determined that Petitioner's testimony regarding psychometrics and related topics, such as item response theory, classical response theory, and high-stakes test reliability and validity, concerned specialized subject matters not within the realm of common knowledge or based on Petitioner's personal perception. Rather, such specialized subject matters required expert witness testimony, pursuant to section 90.702, and as addressed above, Petitioner was not shown to be an expert in any of these specialized subject matters. Because Petitioner's testimony constituted the type of opinion testimony that is not permissible by a lay witness, pursuant to section 90.701, such testimony is not afforded weight in this proceeding. Petitioner testified that "minimal competence," which is the standard measured on the PP Exam, equates to "competence" as defined in the dictionary—that is, the "sufficient ability for a specific need or requirement." However, this position ignores that, for purposes of the Challenged Rule, "minimal competence" is a term of art specifically used, in the psychometric measurement context, to describe the level of competence that corresponds to a passing score on the PP exam. As more fully discussed below, the PP exam is developed and scored using psychometric tools and methods. 8 Nat'l Commc'ns. Indus., Inc. v. Tarlini, 367 So. 2d 670, 671 (Fla. 1st DCA 1979)(lay witness testimony regarding a specialized subject matter was not admissible into evidence because the testimony was not regarding a subject matter about which the witness could testify based on common knowledge or his personal perception.) Petitioner contends that engineering experience is, by itself, a reliable measure of competence, so it is unnecessary to also pass the PP exam. Petitioner testified "the application process and [PP] exam have two entirely different methods to identify the same result: whether an engineer in training is competent enough to become a licensed engineer It is not logical for two checks of competence to come up with different results. There should be consistency." In support of this position, Petitioner relies on section 471.015(2)(a), which requires "at least 4 years of active engineering experience of a character indicating competence to be in 'responsible charge' of engineering." § 471.015(2)(a), Fla. Stat. "Responsible charge" is defined in rule 61G15-18.011(1) as the degree of control an engineer is required to maintain over engineering decisions made personally or by others over which the engineer exercises supervisory direction and control authority. An engineer in responsible charge is the "engineer of record," as defined in rule 61G15-30.002(1). Rule 61G15-30.002(1) defines "engineer of record" as a Florida professional engineer who is in responsible charge. Thus, an engineer who is qualified, for purposes of being in responsible charge pursuant to section 471.015(2)(a) must, in addition to having the minimum statutory experience, be a licensed P.E. This means that he or she necessarily must have passed the PP examination. These statutory and rule provisions collectively reinforce the point that for an engineer to demonstrate competence for purposes of holding himself or herself out as an "engineer," as defined in section 471.005(7), he or she must satisfy all three requirements of section 471.015(2)(a)—i.e., education, experience, and passing the licensing exam.9 9 The requirement to meet these three requirements, including the PP exam, is codified in section 471.051(2)(a). Eliminating the exam requirement and relying strictly on education and/or experience for licensure would require the Florida Legislature to amend this statute Petitioner echoed the testimony of Hoot and Grace that licensure is not a guarantee of competence, and that passing the PP exam does not guarantee minimal competence. To this point, he testified that he does not believe that the PP exam adequately and reliably measures an applicant's ability to practice engineering, and that experience is a better indicator of competence than passing the exam. By way of example, Petitioner described his own experience10—which he characterized as "directly matching" the activities in which a licensed engineer engages—and compared that experience to measuring competence by an exam, which Petitioner characterized as "attempt[ing] to indirectly measure my ability as an engineer." Based on his personal experience, Petitioner contends that experience better demonstrates competence to be licensed as a P.E.; that passing the PP exam does not indicate minimal competence to practice engineering; and that failing the PP exam does not mean that the examinee is not minimally competent. He further testified that examinees who fail the PP exam likely are minimally competent, since the engineering jobs they hold when applying for licensure likely would require that they be minimally competent in order to have been hired.11 to eliminate the exam requirement. The undersigned is not authorized by statute or the Florida Constitution to eliminate the PP exam requirement for licensure under chapter 471. 10 Petitioner's experience, set forth in his P.E. licensure application, was verified by his supervising engineers. 11 Petitioner appears to conflate being determined not "minimally competent" for purposes of passing the PP exam, with "incompetence," which is defined in Florida Administrative Code Rule 61G15-19.001(5) as the "physical or mental incapacity or inability of a professional engineer to perform the duties normally required of the professional engineer." Part of this confusion may be due, in part, to Respondent's response to one of Petitioner's interrogatories asking for a definition of "minimal competence." Rather than directly answering the interrogatory, Respondent referred Petitioner to the definition of "incompetence" for purposes of imposing discipline under Respondent's disciplinary rules—thus causing Petitioner to understandably assume that failing to demonstrate minimal competence through passing the PP exam equates to "incompetence," as defined in rule 61G15-19.001(5). However, the fact that Petitioner has not demonstrated "minimal competence" on the PP exam does not mean that he is incompetent; it simply means that he has not yet passed the PP exam for licensure as a P.E. in Florida. To this point, if failing to demonstrate "minimal competence" by passing the PP exam equated to being incompetence, every person who performs engineering work in To further illustrate this point, Petitioner noted that the data regarding passage rate of the PP exam shows that examinees having zero years of experience are almost twice as likely to pass the PP exam as examinees having 11 or more years of experience. However, as Hood and Grace explained, and as further discussed below, engineers having more than four to five years of experience begin to specialize in narrower fields and "life happens," in that personal and professional circumstances render it more difficult to prepare for and take a high-stakes test. Petitioner also disputed the accuracy of the PP exam preparation and scoring process. In particular, he took issue with the "model law engineer" standard to which the exam is designed. As discussed more extensively below, this standard equates to the competence level of an engineer having four years of engineering experience and who is capable of practicing engineering in a manner that protects the public health and safety. In particular, Petitioner contends that designing the PP exam to the "model law engineer" standard is unfair to anyone taking the exam that does not have exactly four of years of engineering experience. Notably, however, section 471.015(1), which is the statute implemented by the Challenged Rule, establishes four years as the engineering experience required for licensure as a P.E. Thus, the "model law engineer" standard is rationally related to the statutory minimum experience level for purposes of demonstrating minimum competency to be licensed. Petitioner also contends that the subject matters tested on the PP exam are unfairly broad, so that engineers who specialize in a particular area within an engineering discipline—such as specializing in water-related areas in environmental engineering—are disadvantaged by being required to take Florida but has not passed the PP exam would be "incompetent," and, thus, potentially subject to disciplinary action. an exam that covers a broad range of areas beyond his or her area of specialty. Petitioner further contends that it is irrational to test an examinee on particular areas that are irrelevant to his or her work and/or desired career path. However, the PP exam for a particular discipline is specifically designed to ensure that a licensed P.E. is competent to practice over a range of specific areas encompassed within that particular discipline. This is because once a person becomes a licensed P.E., he or she may practice engineering within any discipline or specific area within that particular discipline, subject to the professional and ethical requirements to limit practice to the disciplines and areas in which the engineer is actually competent. Thus, the breadth of the PP exam is designed to help ensure minimal competence to practice engineering in a manner that protects the public health and safety. Petitioner also contends that because the PP exam for some engineering disciplines tests a broader range of areas than the PP exam may test for other disciplines, the exam inconsistently measures minimal competency across the range of engineering disciplines. However, as discussed in detail below, the subject matters tested on the PP exam for a given discipline are chosen by subject-matter experts who are licensed engineers practicing in that particular engineering discipline, and are deemed, by those subject matter experts, to be most important to test for purposes of measuring competency in that discipline. Thus, while the number of discrete subject matters tested on the PP exams may differ across the various engineering disciplines, this difference is, factually and logically, a function of expert consensus regarding which subject matters need to tested to demonstrate minimal competence. Petitioner also contends that the Challenged Rule is vague because it does not specifically identify the disciplines, and the areas within each discipline, that are tested on the PP exam. Respondent has contracted with NCEES to be responsible for preparing, administering, and scoring the PP exams, pursuant to section 455.217(1)(d). NCEES conducts a methodical process, discussed in detail below, to determine the specific disciplines for which to develop a PP exam and the areas to be covered on the PP exam for a discipline. Exam specifications are then developed by subject matter experts within that discipline, and are published by NCEES. These specifications inform prospective examinees regarding the particular areas that will be tested on the PP exam for the discipline, and the number of questions for each specific area that will appear on the exam. Thus, prospective examinees are not left to wonder or guess about which disciplines will be tested; the areas within each discipline that will be tested; or the relative weight that will be assigned to each area tested.12 Although the Challenged Rule does not identify the specific disciplines tested on the PP exam, rule 61G15-21.001(1)(b)—which actually adopts the PP exam as an engineering licensure exam in Florida—states that the PP exam "is given by discipline." Therefore, even if section 455.217(1)(d) required the specific areas of competency to be identified by rule, such areas would have been identified in rule 61G15-21.001(1), rather than in the Challenged Rule.13 12 See Cole Vision v. Dep't of Bus. And Prof'l Reg., 688 So. 2d 404, 410 (Fla. 1st DCA 1997)(a rule is impermissibly vague if it is drafted in terms so vague that men of common intelligence must necessarily guess at its meaning or application. 13 Neither sections 455.217(1)(d) nor 471.015(1) specifically authorize or require Respondent to adopt rules identifying the general areas of competency tested on the PP exam. By contrast, exams developed by the agency pursuant to section 455.217(1)(b) must identify, by rule, the general areas of competency to be tested. Had the Legislature intended for exams authorized under section 455.217(1)(d)—of which the PP exam is an example—to adhere to the same requirement, the statute would have so stated. See Pro-Art Dental Lab, Inc. v. V- Strategic Grp., LLC, 986 So. 2d 1244, 1258 (Fla. 2008)(the specific mention of one thing in a statute implies the exclusion of another). Furthermore, section 120.54(1)(g) expressly requires a rule to address only one subject. Thus, if the Challenged Rule also addressed the areas of competency to be tested on the PP exam, it would violate section 120.54(1)(g). Petitioner also asserted, at the final hearing, that Respondent did not certify the PP exam as meeting the requirements of national examinations and generally accepted testing standards pursuant to department rules, as required by section 455.217(1)(d). However, Petitioner did not raise this alleged invalidity basis in the Rule Challenge Petition, so he is foreclosed from raising and litigating it at the final hearing. See § 120.56(1)(b), Fla. Stat. Petitioner also testified that, in transitioning from paper-and-pencil PP exams to computer-based exams, NCEES is relying on two different theories—classical test theory and item response theory—and that this reliance does not comport with generally accepted testing standards. However, as discussed above, Petitioner was not qualified as an expert in the specialized area of high-stakes examination preparation and scoring; thus, his testimony constitutes lay opinion regarding this specialized subject matter. He did not present any competent substantial evidence to support his contention that the PP exam does not meet generally accepted testing standards.14 Petitioner also testified that item response theory, which is a psychometric tool used in developing and scoring the PP exam, is an invalid means of determining the competence of an engineer. To this point, Petitioner testified that the "model law engineer" is not a real person, but is instead an imaginary person created by subject matter experts to define what a minimally competent engineer should know. Thus, according to Petitioner, 14 Because Petitioner was not qualified, tendered, or accepted as an expert in these specialized subject matters, and because his testimony on these matters consists of inadmissible lay opinion testimony, this testimony has not been afforded weight. §§ 90.701 and 90.702, Fla. Stat. the model law engineer standard is the wrong reference point for determining minimal competency to practice engineering.15 Petitioner also testified that the PP exam does not accurately measure ability, which is a latent trait for which an arbitrary measurement scale must be created. He testified that the model law engineer standard is the midpoint of this scale, and that the purpose of the scale is to determine whether examinees fall above or below that midpoint.16 He further contended that the PP exam does not accurately measure ability, because performance on the exam may be influenced by extraneous variables, such as test anxiety. Petitioner also testified regarding item response theory, which, as previously noted, is a psychometric tool used in developing and scoring high- stakes exams—a subject about which Petitioner had no training in, or knowledge about, until he prepared for the final hearing in this proceeding. Specifically, Petitioner testified that the item characteristic curve is the basic building block of item response theory, and that there are two technical properties of an item characteristic curve: difficulty of the item, and the ability of the item to discriminate between examinees' abilities. Petitioner testified that another basic principle of item response theory is that the examinee's ability is a variable with respect to the items used to determine it. According to Petitioner, this principle rests on two 15 Petitioner's testimony on this point was based on excerpts from a book titled Item Response Theory and a book titled The End of Average. The Item Response Theory book is a treatise on psychometrics, a highly specialized field about which Petitioner was not qualified to testify as an expert, and which is not susceptible to lay witness opinion testimony. Thus, Petitioner's testimony on these points is not assigned weight. See §§ 90.701 and 90.702, Fla. Stat. Additionally, excerpts from The End of Average were determined irrelevant, so were not admitted into evidence. 16 As support for this testimony, Petitioner selectively cited and quoted the deposition testimony of Timothy Miller, Respondent's expert on the development and scoring of NCEES's psychometric-based PP exams. The specific context of Miller's deposition testimony was that when an exam item is overexposed, it is subject to drift, which means that the percentage of correct answers for the item increases to the point that the item no longer is a good discriminator. As further discussed below, Petitioner's testimony on this point was directly and persuasively countered by Miller's expert testimony regarding scaled scores and setting the passing score for the PP exam. conditions: that all items measure the same underlying latent trait, and the values of all item parameters are in a common metric. According to Petitioner, this principle reflects that the item characteristic curve spans the entire ability scale; thus, the practical implication is that a test located anywhere along the ability scale can be used to estimate an examinee's ability, such that an examinee could take a test that is easy or hard, and on average, would score at the same estimated ability level. Petitioner testified that this stands in contrast to classical test theory, which he contends is a better discriminator of examinee ability.17 Petitioner also testified that the psychometric methods used to develop and score the computer-based PP exams are flawed because "difficulty" is subjective and entirely dependent on the individuals developing the PP exam. Thus, according to Petitioner, in scoring a computer-based multiple-choice PP exam, it is impossible to know whether a particular examinee got the answer right due to a reasonable approach in answering the question, or by guessing. Petitioner contends that for this reason, multiple-choice test questions developed using item response theory are not good discriminators of examinees' ability; thus, even if an examinee does not correctly answer enough questions to pass the exam, that does not mean that the examinee is not knowledgeable in that area.18 Petitioner further testified that because difficulty is a subjective parameter, different forms of the PP exam inherently have different levels of difficulty. Thus, according to Petitioner, it is a matter of luck whether an examinee takes a more difficult form or an easier form of the exam. Further to this point, Petitioner testified that because an examinee does not take multiple forms of the exam, but instead takes only one form, the determination of the examinee's ability is solely dependent on a subjective 17 Refer to note 15, supra. 18 Refer to note 15, supra. parameter—i.e., the difficulty of the test questions as determined by subject matter experts. Petitioner contends that, as a result, the PP exam does not accurately measure an examinee's ability, and, therefore, is not a valid exam.19 Petitioner also testified that because statistical indices of reliability and validity are not attributes of an exam, a researcher may select what seems to be an appropriate test for his or her purposes, when, in fact, the selected test does not have any level of reliability or validity. Thus, Petitioner testified, reliability and validity are values that reside in test scoring, not in the test itself. Petitioner testified that validation, in statistics, is the process of accumulating evidence that supports the appropriateness of the inferences that are made of student responses for assessment uses. He testified that validity refers to the degree to which the evidence indicates these interpretations are correct and the manner in which the interpretations are used is appropriate.20 Petitioner testified regarding three types of validity evidence: content, construct, and criterion evidence.21 Specifically, Petitioner testified that content evidence refers to the extent to which an examinee's responses to a given assessment reflect the examinee's knowledge of the content being tested; thus, to the extent an exam inadvertently measures a parameter that is not related to the examinee's knowledge of the content being tested, it is invalid. 19 Refer to note 15, supra. 20 Petitioner's testimony relied on, or was paraphrased from, a document titled The Scoring Rubric Development. Again, because this topic and document address a matter within the specialized fields of psychometrics, high-stakes testing, and test-scoring statistics, which are areas in which Petitioner was not qualified as an expert, and which are not susceptible to lay opinion testimony, pursuant to sections 90.701 and 90.702, Petitioner's testimony relying on this document, including his testimony regarding content, construct, and criterion-related evidence, is not afforded weight. 21 Refer to note 15, supra. This determination regarding the weight afforded Petitioner's testimony applies to paragraphs 94 through 100 herein. According to Petitioner, the content-related evidence for the PP exam for each discipline is inconsistent, so that the PP exam for a given discipline does not accurately measure minimal competence for that discipline. Petitioner also testified that the weighting of different topics on the PP exam necessarily creates an advantage for engineers who work in areas more heavily weighted on the exam, while creating a disadvantage for engineers who work in areas that are less heavily weighted on the exam. Petitioner also testified that to accurately determine minimal competence in all engineers, the model law engineer standard should be keyed to, and the content tested on the exam should be directed toward all engineers, including those having more than four years of experience. Petitioner also testified that construct-related evidence consists of external benchmarks, such as results and explanations, of internal evidence of psychological processes, such as reasoning. Petitioner testified that because multiple-choice exams do not provide evidence of an individual's reasoning process, they do not generate construct-related evidence for purposes of determining exam validity. Petitioner testified that free response paper-and pencil-exams provide construct-related evidence, so are better indicators of an examinee's knowledge. Petitioner also testified regarding criterion-related evidence, which relates to the extent to which the results of an assessment, such as the PP exam, correlate with a current or future event. By way of illustration, Petitioner testified that criterion-related evidence considers the extent to which a student's performance may be generalized to other relevant areas. Petitioner testified that an examinee's performance on the PP exam is not generalizable to other relevant activities, so it is impossible to determine whether the exam actually corresponds to minimal competence in the workplace. In sum, Petitioner testified that the PP exam does not meet content, construct, or criterion-based evidence for purposes of determining whether it is a valid exam. Thus, Petitioner reasons, it is logical to conclude that because the PP exam is not a good discriminator between minimally competent and incompetent engineers, it does not reliably and adequately measure competence. Petitioner also testified that because passing the PP exam is only one component of licensure, it fails to meet criterion-based validity, in that the exam, by itself, does not certify a passing examinee to practice as a P.E. As Petitioner put it, "you're just passing the exam as part of the requirement for licensure." Petitioner reasoned that if passing the PP exam corresponds to minimal competence, the experience and education requirements of section 471.015(2)(a) are redundant. Also to this point, Petitioner testified that the preapproval process to take the PP exam is directly related to an examinee's actual work experience as an engineer, while taking the exam merely entails answering questions about engineering work. Thus, Petitioner contends, a competent engineer, as shown through Respondent's preapproval process, may nonetheless fail the exam. Petitioner asserts that this further shows that the exam does not accurately measure minimal competence. Petitioner also testified that, in his view, delaying licensure of potentially competent engineers due to postponing the exam due to the Covid-19 pandemic does not serve the interest of public health and safety. To that point, he testified that the inability to obtain a variance, which would relieve examinees from having to take and pass the exam under such circumstances, renders the Challenged Rule arbitrary.22 Petitioner also contended that passing the PP exam should not be required, because other engineering professional associations—specifically, the European Federation of National Engineering Associations ("FEANI"), which represents engineers in European countries—allow licensure through 22 Petitioner's point regarding inability to obtain a variance or waiver is addressed in the Conclusions of Law. education and experience requirement, without requiring a professional exam to be taken and passed. However, because section 471.015, which governs the licensure of engineers in the state of Florida, requires a professional licensure examination to be taken and passed as part of the P.E. licensure requirements, Petitioner's testimony and argument regarding FEANI's practices and requirements take issue with the statute, rather than the Challenged Rule, and, thus are irrelevant. Petitioner also contends that the examination fee for the PP exam is arbitrarily set, rendering the Challenged Rule arbitrary. However, as discussed above, the Challenged Rule only addresses determining the passing grade for the PP exam using psychometric methods. The Challenged Rule has nothing to do with establishing or setting an examination fee. Thus, this challenge ground has no basis in fact or law.23 C. Findings Regarding the Evidence Presented in Respondent's Case Respondent's Expert Witnesses Respondent presented the testimony of Timothy Miller, P.E., who serves as Director of Examination Services for NCEES. Miller has held this position for approximately 15 years. His job-related activities and responsibilities include directing exam development, publication, scoring, and fulfillment of the licensing exams for engineers and surveyors; coordinating exam development committees consisting of over 800 volunteer subject matter experts who work on developing each NCEES exam; overseeing the exam development process and providing advice and guidance regarding engineering exam development, administration, production, scoring, analysis, and reporting; serving as a testing process consultant to exam development committees; and other exam-development and administration-related matters. 23 Additionally, this challenge ground was not raised in the Rule Challenge Petition, so is not at issue in this proceeding. See § 120.56(1)(b), Fla. Stat. Before Miller was promoted to his current position, he served as an examination development engineer for NCEES. In that position, Miller was responsible for planning and coordinating engineering exam development, production, administration, scoring analysis, and reporting for certain assigned examinations; serving as a testing consultant working with engineering exam development committee chairs regarding quality and number of exam development volunteers; and overseeing development and administration of the licensing exams in the specific fields of environmental controls systems, metallurgical engineering, and mechanical engineering. Through his experience in these positions with NCEES, Miller is an expert in professional examination development and scoring, particularly with respect to the development and scoring of the NCEES FE and PP examinations. Before being employed with NCEES, Miller practiced civil and structural engineering with several private-sector engineering firms. He has been a professional engineer since 1984, and is licensed as a P.E. in South Carolina, North Carolina, Maryland, Delaware, Pennsylvania, and New Jersey. Respondent also presented the testimony of Dr. Michelle Rodenberry, P.E., an associate dean and associate professor at the Florida A&M University–Florida State University College of Engineering. Her engineering expertise is in the field of structural engineering—specifically, bridge engineering. Rodenberry was appointed to the Board in 2012, and she served as a Board member until 2018. She is now an emeritus Board member. While on the Board, she served as chair of the education committee, and was involved in reviewing applications for licensure as a P.E. in Florida. Development, Scoring, and Validation of the PP Exam The NCEES engineering exams are national licensing exams that are recognized by every engineering licensing entity in each of the U.S. states, as well as by the engineering licensing entities in Washington, D.C.; Puerto Rico; the U.S. Virgin Islands; and the other U.S. territories and protectorates. There are approximately 26 different engineering disciplines, each of which is tested by a separate PP exam specific to that discipline. In the 1990s, NCEES decided to transition from subjectively-graded pencil-and-paper examinations to an objectively-graded computer-based multiple-choice exam format. Currently, approximately one-third of the PP exams, including environmental engineering, have been converted to a computer-based format, and all but one of the exams in the other disciplines is in the process of being converted. The reason NCEES is transitioning the PP exam from a pencil-and- paper format to a computer-based format consisting of multiple-choice questions is to help eliminate subjectivity in grading, so that the exam papers are consistently graded across groups of examinees. Additionally, a computer-based format consisting of objective multiple-choice questions allows the difficulty of the exam to be psychometrically evaluated for purposes of determining the passing score for a particular administration of the exam. To that point, because computer- based multiple-choice exams are objectively scored, exams offered at different times during the year are able to be compared, or equated, for purposes of setting the passing grade for a particular exam administration.24 Respondent entered into a contract with NCEES in 2009, pursuant to which NCEES provides the FE and PP exams for engineer licensure in Florida. In 2013, the contract was amended to allow NCEES to provide the exams by computer-based testing, using Pearson Vue as its exam 24 As Miller explained,"[i]f they were different on a difficulty level, the harder exam, the standard would actually be lowered so that it would be fair across administration so everybody was treated consistently. Or if my exam was less difficult, the standard would be raised. I would have to get more questions right." administering entity. The FE and PP exams are administered by Pearson Vue at its testing centers. NCEES develops model laws and rules that represent best practices with respect to state licensing of engineers. The aim of these model laws and rules is to achieve uniformity and consistency throughout the states and the U.S. territories and protectorates in the licensure of professional engineers. A significant benefit of such consistency and uniformity is the resulting "mobility" for licensed professional engineers—that is, the ability to more easily become licensed to practice engineering in multiple states. The NCEES model laws and rules establish the "model law engineer," which defines and constitutes the standard for minimal competence in a specific engineering discipline for purposes of being licensed as a P.E. in that discipline. The model laws and rules define the "model law engineer" as a person who holds a degree from an engineering educational program accredited by ABET, has four years of active engineering practice experience, and passes the FE and PP exams. The model law engineer standard equates to the competence level of an engineer having four years of engineering experience who is capable of practicing engineering in a manner that protects the public health and safety. This constitutes the minimum competence level that an applicant must demonstrate for purposes of being licensed as a P.E.25 in the 50 states and the U.S. territories and protectorates. Thus, the NCEES PP exam is constructed to test engineering ability keyed to the model law engineer standard. That is, the PP exam is designed to determine the ability level of an applicant for P.E. licensure for purposes of 25 Refer to note 12, supra. The term "minimal competence," as used in the Challenged Rule, is specifically keyed to the "model law engineer" standard for purposes of being licensed as a P.E. It is not meant to indicate or imply that an engineer who does not take or pass the PP exam is per se incompetent, such that he or she is not competent to engage in work constituting engineering, as defined in section 471.005(7). comparing that ability level to that of an engineer having four years' experience who is able to practice engineering in a manner that protects the public health and safety. Examinees having four years of engineering experience after graduation have the highest pass rates on the PP exams. Pass rates for examinees with more or less than four years of experience are lower, typically in proportion to the length of time before or after the four-year experience mark when they take the PP exam. Miller explained, credibly and persuasively, that the reason for the drop-off in PP exam performance after the four-year mark is that "life happens." Engineers gain more experience, and many become specialized in a relatively narrow niche, or move into managerial, non-technical positions. Additionally, because the PP exam does, in part, test subjects that one learns in college, the longer an examinee is out of college, the less subject matter recall in certain areas he or she may have. "Psychometrics" is the specialized field of study concerned with the theory and technique of psychological measurement. Specifically, psychometrics entails the objective measurement of skills and knowledge, abilities, and educational achievement. Among other specialized areas of practice, psychometricians focus on the construction and validation of assessment instruments, and theories, such as item response theory, that relate to psychological measurement. Psychometricians typically have graduate training and all possess specialized qualifications that enable them to engage in objective psychological measurement. PP exams are designed to determine minimal competence in a specific engineering discipline. "Minimal competence" is the minimal amount of knowledge required to practice in that particular engineering discipline in order to protect the public health and safety. For any specific engineering discipline for which it has been determined that a PP exam should be given,26 there is an approximately three-year due diligence period in which subject matter experts in that discipline work to determine the topics that should be tested on the exam. The PP exam for each specific discipline is developed by subject matter experts, who volunteer and meet on a monthly basis to develop, review, and evaluate the questions for the PP exam for that specific discipline.27 The process of determining which topics should be tested on a PP exam, termed the Professional Activities and Knowledge Study ("PAKS") process, is a standard practice used to determine the specific topics to be tested on a PP exam. As part of the PAKS process, a consulting psychometrician28 employed by Pearson Vue; 15 to 20 engineers who are licensed in another engineering discipline; and subject matter experts who may teach a particular engineering discipline for which the PP exam is being developed, work together to develop consensus regarding the specific topics that engineers having four years of experience practicing in that discipline need to know in order to safely practice engineering in a manner that protects the public health and safety.29 26 For a PP exam to be developed for a new engineering discipline, at least ten NCEES- member state engineering boards must request that such exam be developed, and at least one ABET-accredited program in that specific discipline must exist. 27 Over the years of development and administration of the PP exam, hundreds of licensed engineers have provided input regarding the topics that should be, and are, tested in each discipline and the relative weight given to each topic on the PP exam. 28 Pearson Vue's psychometricians who develop, score, and evaluate NCEES's exams have Ph.D. degrees in psychometrics or statistical analysis. 29 Subject matter experts selected to develop the PP exam questions are chosen based on consideration of the type of practice, such as governmental and private practice; gender; ethnicity; length of time of licensure as a P.E.; and geographic considerations. All subject matter experts must be licensed as a P.E. by a state engineering licensure board in order to The consulting psychometrician builds a questionnaire that lists the specific topics identified by the PAKS committee, and distributes an online survey to engineers who practice in the discipline for which the PP exam is being developed. The survey seeks input regarding the relative importance of each specific topic for purposes of testing to demonstrate minimal competence in the discipline. Based on the survey responses from engineers practicing in the discipline, exam specifications are developed. The exam specifications identify each specific topic to be tested on the PP exam, and the number or percentage of exam questions that will address each specific topic within that discipline. The exam specifications must be approved by an oversight committee. Once the exam specifications have been approved, the subject matter experts for that specific engineering discipline for which the PP exam is being developed prepare the PP exam questions—also termed "items"—and review and evaluate them for clarity, demographic neutrality, and other parameters, so that the items will reliably and validly test engineering ability. In computer-based multiple-choice PP exams, the questions are prepared such that for each question, there is only one correct answer and three other plausible, but incorrect, alternative choices. The individual exam questions are reviewed numerous times by the subject matter experts before they are moved into an exam question bank for use on the PP exam. Once the exam questions have been developed and banked for use on a PP exam, a standard-setting committee, consisting of ten to 15 licensed engineers having diverse backgrounds, experience, and demographic features, reviews the exam to determine the minimum passing score—or "cut score"—on the exam. The cut score equates to the ability level of an engineer serve in this capacity. As noted above, over the years of development and administration of the PP exam, hundreds of engineers have provided input in developing each PP exam. having four years of experience who is minimally competent to practice engineering at a level that protects the public health and safety. This method of using subject matter experts to examine the content of each exam question and predict how many minimally-qualified examinees would answer each question correctly is termed the "Modified Angoff Method." The standard-setting committee then takes the exam, and the psychometrician analyzes the data from the committee's exam sitting. Using this data and analysis, the standard-setting committee then reviews, and reaches consensus, regarding each question, for purposes of determining the proportion of minimally competent engineers who would answer that specific question correctly. Based on the information generated by this process, the psychometrician develops the "panel recommended passing score," with a statistical margin of error. The psychometrician presents this recommended passing score to a committee of five persons, consisting of two state licensing board members and three subject matter experts who observed the exam development process. Based on the psychometrician's recommendation, the committee makes the final decision regarding the minimum passing score for the exam. Each PP exam question is developed and evaluated using the process described above, and is placed in bank for use on a PP exam. The psychometrician uses item response theory to "calibrate"—i.e., determine the relative difficulty level of—each exam question.30 An exam question is not banked for use on future sittings of the exam unless it has had at least 200 responses on a previous exam, so that statistics for each item's performance can be generated for purposes of item calibration. 30 Item response theory is one of many psychometric methods, or tools, used to weight exam questions for purposes of creating different forms of exams having the same level of difficulty. Depending on the specific discipline, a question bank for a PP exam may consist of many thousands of questions.31 Using item response theory, the psychometrician converts the passing score to create a scale from -5 to +5, which will equate to the examinee's ability level as measured by the exam. Once the passing score for the PP exam has been determined, different PP exam "forms" are created for administration in different exam sittings. Exam "forms" are essentially different versions of the PP exam that consist of different individual questions of the same difficulty level, as determined using item response theory, for each specific topic on the exam. Thus, if a PP exam was administered, for example, in April and October, the different exam administrations would consist of different forms—meaning that the exam would consist of different questions, but the questions would be of the same difficulty level for a specific topic tested on the exam.32 Additionally, because exam item difficulty has been calibrated using item response theory, different forms of a PP exam can be given during the same exam administration at different locations.33 Importantly, because the difficulty of the exam items has been calibrated using item response theory, the different exam forms are statistically equivalent in difficulty. 31 The only items that will be used on the graded portion of the PP exam are questions that have known statistics such that they have been calibrated for difficulty. However, there may be other "pretest" questions on the exam strictly for purposes of gathering statistics regarding performance on the questions for potential inclusion in the exam item bank; these "pretest" exam questions are not graded for purposes of determining the examinee's score on the exam. 32 As Miller explained, for an administration of an exam at different locations at the same time, the form administered at a particular location consists of different questions than the form administered at another location; however, the exam forms are equivalent in terms of the number of questions addressing a particular topic and the difficulty of the items addressing that topic. 33 Using the "linear-on-the-fly" ("LOFT") method to generate different forms of the exam also helps ensures exam security, since persons sitting near each other during an administration of the exam will not have the same exam form. As a result of using these processes, including the Modified Angoff Method, and applying item response theory to calibrate the exam items for purposes of constructing different, but statistically equivalent, forms of the PP exam, examinees are not graded on how they perform against each other, but instead are graded against the cut score set for the exam. To ensure that different forms of the exam are statistically equivalent in difficulty, Pearson Vue uses the LOFT method,34 which employs an algorithm to ensure that, across all of the exam forms, all examinees get the same number of questions of the same level of difficulty on the same topics. The algorithm randomly generates, or assembles from banked exam questions, different exam forms based on the exam specifications (i.e., the specific topics tested and relative weight/number of exam questions for that topic) and the difficulty level of the exam questions, such that the different exam forms generated by LOFT are statistically equivalent to each other. Using item response theory to calibrate specific exam question difficulty based on the statistical probability of being answered correctly enables examinees taking different, but statistically equivalent, forms of the exam to be compared to the passing standard for purposes of determining whether they pass the exam. Thus, examinees are compared to an ability level—here, minimal competence—rather than to each other.35 This method ensures that all examinees take an exam of equivalent difficulty, which, in turn, helps ensure the fairness of the exam. 34 For the engineering disciplines having too small a group of examinees to employ item response theory or LOFT to generate different exam forms, each examinee takes the same exam instead of taking different forms of the exam, and the exam typically is offered only on one day, rather than multiple days, per year. 35 By way of example, Miller explained that if two examinees each answer five questions on the same topic on the exam, and one examinee answers four easier questions correctly and the other examinee answers two comparatively more difficult questions correctly, the examinee answering the two more difficult questions correctly may have a higher ability level on that particular topic, due to the comparative difficulty of the questions that examinee answered correctly. Once a PP exam is administered, Pearson Vue scores the examination and sends NCEES the information regarding whether each examinee has passed or failed the exam. Pearson Vue also provides each examinee the information regarding his or her performance on the exam compared to the minimum competence standard. The examinee's performance is expressed as a scaled score, for each specific topic tested on the exam, and for the entire exam. Specifically, using psychometric statistical methods, the ability level of the examinee is expressed as "theta," and is placed on a scale of 0 to 15 for each of the specific topics tested. The examinee's overall theta across all specific topics tested is then compared to the "minimal competence" passing standard, which is also expressed as a scaled score using the same 0 to 15 scale. After an exam is taken and scored, the consulting psychometrician analyzes this data, called "response data," for each exam question, for each examinee, to calibrate the items for purposes of determining whether a particular question performs well in discriminating ability level of the examinees. The psychometrician may recommend that an exam question be "retired" because it is not performing as a good discriminator of ability level. Examples are where an exam question is too difficult or too easy, such that it does not discriminate well in determining ability level; where an item takes too long to answer or is ambiguous; where an item has been "overexposed" by having become publicized such that future examinees have access to the question and scores on the question become high; or where an exam contains "bad pair" items, such that the answer to one item may suggest, or lead to, the answer on another similar item. Having a psychometrician involved in tracking and analyzing exam data enables such circumstances and situations—which may influence the scores on a test item for reasons not related to the examinee's ability—to be identified and corrected. Returning to a free response, paper-and-pencil exam format for the P.E. licensing exam would provide a far less objective, fair, and accurate measure for determining minimal competence for purposes of being licensed to practice engineering. It also would negatively affect the ability of licensed engineers to become licensed in other states. Due to the use of psychometric methods in developing and scoring, the PP exams are very reliable across multiple administrations of the exam— to the point that all of NCEES's psychometric-based PP exams score upwards of .9 on a scale of 1.0.36 Psychometric methods, including item response theory, are used in developing, administering, and scoring many different types of high-stakes professional and academic examinations, including medical school admissions examinations, and nursing, medical examiner, internal auditor, and architecture licensure examinations. NCEES audits approximately one-third of the exams administered by Pearson Vue on an annual basis, to independently evaluate the accuracy of the psychometric services provided by Pearson Vue, and to ensure that the exams have been created pursuant to NCEES's guidelines, procedures, and requirements. NCEES also retains independent psychometricians to review Pearson Vue's exam-related reports and analyses, to ensure that Pearson Vue is following standard psychometric rules of good practice. In sum, the use of objective psychometric methods, including the methods discussed above, to develop, score, and evaluate the PP exam ensures that minimal competence, for purposes of licensure as a P.E., is accurately measured by the exam. Role of the PP Exam in Licensure of PEs in Florida As discussed above, to be licensed as a P.E. in Florida, an applicant must have a college degree from an ABET-accredited institution, four years of 36 Test reliability refers to the degree of consistency with which a test measures a particular subject matter across different administrations of the test. A test has a high reliability score if it consistently produces similar results under consistent conditions. A 1.0 reliability score reflects perfect consistency in results across different administrations of a test. An acceptable reliability score target for high-stakes tests is .7 or higher. active experience in engineering practice, and have passed the FE and PP exams. Thus, the P.E. exam is a vital component of determining that an engineer licensed as a P.E. to practice in Florida is able to practice at a competence level that protects the public health and safety. Unlike the education and experience requirements for licensure— both of which may entail a great deal of variability in quality and breadth across applicants—the PP exam constitutes an objective, consistent tool37 to measure an applicant's level of competence for purposes of determining whether the applicant possesses the minimal competence needed to practice engineering in a manner that protects the public health and safety. As such, the PP exam constitutes a uniform measure of minimal competency for purposes of licensure as a P.E. in Florida. As discussed above, this does not mean that a person who engages in engineering work but has not passed the PP exam is incompetent; it merely means that he or she has not demonstrated minimal competency on this required objective measure of competency for licensure purposes. As discussed above, the PP exam is specifically designed to ensure that a licensed P.E. is competent to practice over a range of specific areas encompassed within a particular discipline. This is because a licensed P.E. is authorized to practice engineering within any discipline or area, subject to professional and ethical standards. The breadth of the PP exam thus helps ensure minimal competence to practice engineering in a manner that protects the public health and safety. 37 This consistency and uniformity is the direct result of the psychometrically-based exam development, scoring, calibration, and validation methods discussed above. The purpose of P.E. licensure is to inform and protect the public, which is entitled to rely on such licensure as indicating that the licensee is competent to practice engineering.38 Administration of the PP Exam During the Covid-19 Pandemic As discussed above, Petitioner has alleged that the Challenged Rule is arbitrary on the basis that it does not address contingencies for offering the exam if unforeseen circumstances prevent regular administration of the PP exam. Specifically, Petitioner points to the fact that Pearson Vue cancelled the April 2020 PP exam administration due to the Covid-19 pandemic. Pearson Vue has rescheduled the PP exams for various times and at various locations around the country in an effort to make the PP exam available for prospective examinees during the pandemic.39 Pearson Vue also is taking substantial steps to protect persons who have applied to take the PP exams during the Covid-19 pandemic. To that point, Pearson Vue has retrofitted its testing centers to help ensure the safety of the examinees as they sit for the PP exam. Specifically, the number of examinees who will be in a testing room has been reduced; masks are required to be worn by examinees and proctors; testing stations are cleaned between each use; some additional test center locations have been added; and some states have relaxed rules to allow examinees to sit for the exam in states other than the one for which they are applying for licensure. At present, the exams are not able to be offered over the internet so that examinees are able to take the exam at a remote location. A substantial reason for this is lack of exam security, which is necessary to protect and 38 As noted above, a person does not have to be licensed as a P.E. to engage in engineering work in Florida. However, if a person wishes to hold himself or herself out to the public as a P.E., then that person must satisfy the requirement to pass the PP exam, which is an indicator of minimal competence for purposes of licensure. 39 Among other things, Petitioner alleges, in paragraph 19 of the Rule Challenge Petition, that the Challenged Rule is invalid under section 120.52(8)(d) because it does not address circumstances where an examination cannot be administered due to force majeure. As maintain the exam's integrity. Additionally, the lack of guarantee of internet service reliability and functionality for every examinee is a crucial consideration, since failed internet connections could significantly affect the fairness of the exam.
The Issue Whether Petitioner is entitled to additional credit for his solutions to three problems on the Principles and Practice of Engineering portion of the engineering licensure examination administered on October 29, 1999, by the National Council of Examiners for Engineers and Surveyors.
Findings Of Fact Based upon the evidence adduced at hearing and the record as a whole, the following findings of fact are made: On October 29, 1999, as part of his effort to obtain a Florida engineering license, Petitioner sat for the Principles and Practice of Engineering Examination (Examination). This is a national examination developed and administered by the National Council of Examiners for Engineers and Surveyors (NCEES). Petitioner chose to be tested in mechanical engineering. Petitioner received a raw score of 47 on the Examination. For the mechanical engineering specialization, a raw score of 47 converts to a score of 69. To pass the Examination, a converted score of 70 is needed. Petitioner formally requested (in writing, by letter dated March 13, 2000) that his solutions to Problems 141, 144, and 147 on the Examination be rescored. Petitioner's written request was made to the Board's "Legal Section," which forwarded it to the NCEES. The NCEES's rescoring of Petitioner's solutions to Problems 141, 144, and 147 resulted in his receiving no additional points. The Board received the NCEES's rescoring results on or about April 25, 1999. After receiving a letter from Petitioner (dated May 3, 2000) requesting a "formal hearing," the Board referred the matter to the Division. Problems 141, 144, and 147 were worth ten raw points each. Petitioner received four raw points for his solution to Problem 141. In his solution to Problem 141, Petitioner failed to take into consideration bending stresses and loads. Therefore, in accordance with the requirements and guidelines of the NCEES scoring plan for this problem, the highest raw score that he could have received for his solution to this problem was a four, which is the score he received. Petitioner received a raw score of two for his solution to Problem 144. In rescoring Petitioner's solution to this problem, the NCEES rescorer made the following "comments": A correct solution [to this problem] must include an energy balance on the open feedwater heater to determine the fraction of flow through turbine T1 that is extracted and taken to the open feedwater heater. a correct equation for determining the specific work developed by the two turbines on the basis of one pound entering turbine T1. The equation the examinee has written assumes the same flow through both turbines. determination of the mass rate of flow (m1) at the inlet to turbine T1. This is determined by dividing the net power by the specific net work. determining the rate at which heat is added in the steam generator and reheater. finally, dividing the rate at which heat is added in the steam generator by the heating value times 0.75 with the appropriate conversion factors. The examinee has used the new power (200 MW or 200 x 105)as the rate at which heat is added in the steam generator and reheater. This is incorrect. The scoring plan states 2 RUDIMENTARY KNOWLEDGE . . . OR-(3) determines tons/day = Wnet/7650, Wnet = (h1 - h2) + (h3 - h4) This is what the examinee has done. Based on the scoring plan and the above analysis, a score of 2 is recommended. There has been no showing that the foregoing "analysis" was in any way flawed or that application of the requirements and guidelines of the NCEES scoring plan for this problem should have resulted in Petitioner receiving a raw score higher than two for his solution to Problem 144. Petitioner received a raw score of four for his solution to Problem 147. In rescoring Petitioner's solution to this problem, the NCEES rescorer made the following "comments": The examinee used an incorrect temperature difference in [his] calculation of the heat transferred by convection and radiation from the outer surface of the pipe. Most of the examinee's work for requirement (b) was not needed. In doing that unnecessary work, however, [he] made two significant errors: 1. [He] evaluated a radiation exchange between the steam inside the pipe and the environment surrounding the pipe. The pipe shields the environment surrounding the pipe from the steam. 2. The examinee's equation "Total heat Loss = Conductive + Radiation" is not satisfactory. In attempting to evaluate the heat transfer from the insul[a]ted pipe, [he] assumed that the outer surface heat transfer coefficient was very high; 3.0 is not high. The examinee made no attempt to evaluate the payback period for the insulation. There has been no showing that the foregoing analysis was in any way flawed. For the errors made by Petitioner in his solution to Problem 147, a 50% "grade reduction" was warranted pursuant to the "error analysis" portion of the NCEES scoring plan for this problem. 1/ The remaining portions of the scoring plan for Problem 147 provided as follows: 10: Essentially complete and correct solution. May have one or two minor math, data, or chart reading errors. . . . Grade of 8: A grade of 8 will result from having any combination of the above listed errors which causes a grade reduction between 10% and 50%. A Grade of 6: A grade of 6 will result from having any combination of the above listed errors which causes a grade reduction between 30% and 50%. Grade of 4: 2/ A grade of 4 will result from having any combination of the above listed errors which causes a grade reduction between 50% and 70%. Grade of 2: A grade of 2 will result from having any combination of the above listed errors which causes a grade reduction between 70% and 90%. Grade of Zero: Nothing presented that warrants a grade of at least 10%. It is unclear from a reading of the NCEES scoring plan for Problem 147 whether a grade reduction of 50% should result in a raw score of four or six. The plan is ambiguous in this regard. While it may be reasonable to interpret the plan as requiring that a raw score of six be given where there is a grade reduction of 50%, the plan is also reasonably susceptible to the interpretation that a 50% grade reduction should result in a raw score of four, the score Petitioner received for his solution to Problem 147. It therefore cannot be said that the scoring of his solution to this problem was inconsistent with the problem's scoring plan, as reasonably construed.
Recommendation Based on the foregoing Findings of Fact and Conclusions of Law, it is RECOMMENDED that a final order be entered rejecting Petitioner's challenge to the failing score he received from the NCEES on the Principles and Practice of Engineering portion of the October 29, 1999, engineering licensure examination. DONE AND ENTERED this 13th day of October, 2000, in Tallahassee, Leon County, Florida. STUART M. LERNER Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 SUNCOM 278-9675 Fax Filing (850) 921-6847 www.doah.state.fl.us Filed with the Clerk of the Division of Administrative Hearings this 13th day of October, 2000.
The Issue Is Petitioner entitled to credit for her answers to any of the questions she specifically challenged or for the four questions deleted by the Department of Environmental Protection (Department) on the February 1998 Class "B" Domestic Wastewater Operator Certification examination (wastewater examination)? Was the Department's administration or grading of Petitioner's examination arbitrary, capricious or otherwise unfair so as to entitle Petitioner to either additional points for a passing grade or an opportunity to retake the examination without cost?
Findings Of Fact Petitioner took the Class "B" wastewater operator certification examination given in February 1998 by the Department of Environmental Protection. Pilot questions are questions submitted by licensees and educators which do not yet have a "performance record" of testing validity. Petitioner alleged that pilot questions were used on her examination contrary to rules of the Department. Petitioner submitted no competent evidence to establish this allegation and it was credibly refuted. There are no pilot questions in the bank of potential questions from which the examination was composed. Petitioner conjectured that questions on her "B" level examination may have been drawn from a bank of questions for a higher level ("A") certification examination. In fact, the examination questions for the "B" level examination were selected from a bank of questions developed by the Department of Business and Professional Regulation. The Department of Business and Professional Regulation was the agency that had jurisdiction over the operator certification program before the Department of Environmental Protection assumed responsibility therefor. The selection of the examination questions was accomplished by selecting the percentage of questions from a range in a subject area already predetermined by rule and a computer program inserting the number of questions to fill that percentage. There is no way the computer program can select questions from another level of examination, for instance "A" level or "C" level. Prior to the examination, candidates for examination were advised they would have four hours to complete the examination. Examinees for the February 1998 examination in fact were provided four full hours after all preliminary matters and reading of instructions were completed. Prior to the examination, the Department provided candidates for examination with a list of subject areas that it intended to place on the examination, so that candidates could effectively prepare. All subject areas, except one, were in fact covered on the February 1998 examination. In some instances, a single question/answer satisfied two or more subject areas because of content equally applicable to each subject area. In other instances, the same subject area was covered by several questions/answers. Only one subject area that was listed in the pre-examination information did not appear on the February 1998 examination. That subject area was "energy." The reason that the subject area of "energy" was not included on the February 1998 examination was that there were no energy questions in the bank of questions which the Department of Environmental Protection had inherited from the Department of Business and Professional Regulation. The sole result of the absence of any energy question on the examination is that Petitioner and all other examinees in her group did not receive as thorough an examination in a single subject matter area as the licensure board had aspirationally intended. However, all examinees were equally treated. Originally, there were 87 questions on the February 1998 examination. After the examination was administered and a special analysis report on the grades was produced, the Department's examination review committee met with the examination consultants. The committee recommended to the Department, and the Department accepted the recommendation, that four questions should be deleted because they were misleading or for some other reason failed to adequately and reliably measure the examinees' ability to practice at a Class "B" license level. Removal of the four questions only lowered the mean score by one point, thereby creating a slightly easier examination while simultaneously slightly increasing its reliability. Examinees were instructed to select the best multiple choice answer for each question. Each of the questions was equally weighted. The Department interpreted Rule 61E12-41.005(5), Florida Administrative Code, as requiring that examinees achieve at least a 65% rounded score on the examination in order to pass the examination. In order to determine an examinee's success on the examination, the Department multiplied the initial 87 questions by 65% (.65) and so determined that an examinee would need at least 54 correct questions/answers to earn a passing grade. In determining a candidate's grade on an operator licensure examination, the Department determines the number of correct answers needed to reach the minimum rounded score of 65%. A special analysis report also indicates how many correct answers equal each percentage grade. If this number is not a whole number, the Department uses the rounding method to reach a whole number, based on 0.5 +/- percentage. By the foregoing grading interpretation, before deletion of the four questions, Petitioner's rounded score was 60%, with 52 correct answers. Petitioner's grade improved with the deletion of the four questions, because she had incorrectly answered each of the four questions which were later deleted. After the four questions were deleted, the same grading system resulted in a rounded score of 63% with 52 correct answers. By letter dated April 6, 1998, the Department notified Petitioner that she had failed the examination because she did not get a rounded 65% score based on 52 correct answers. After receiving the letter, Petitioner requested a review of the examination. Petitioner was allowed to review the questions and answers she had missed. Petitioner was also allowed to write comments on the question sheets which she reviewed. Petitioner's comments were submitted to the examination review committee of the Department for the committee's review. Upon review of Petitioner's comments and the examination, the committee determined that the questions and required answers were accurate and fair. It recommended no change to Petitioner's score. Petitioner was notified that no change would be made to her score. Petitioner then timely requested an administrative hearing. Although Petitioner's two letters/petitions (see Preliminary Statement supra.) initially raised issues concerning a number of examination questions, Petitioner only presented evidence concerning the contents of question 78 at formal hearing. Question 78 tested examinees' knowledge of appropriate emergency response activity and notification concerning the release of chlorine gas. Petitioner asserted that question 78 was vague, ambiguous, and misleading because it did not specifically state that a "reportable quantity" was to be considered in choosing the best answer from among multiple choice options of reporting a chlorine spill to one entity, two entities, three entities or no entities. For this reason, Petitioner alleged that her answer could have been an answer which was equally correct ("multi- keyed") with the answer selected as correct by the Department. Mr. Dawkins, who was accepted as an expert in emergency response and community right-to-know, testified that the question was not misleading. Mr. Dawkins is not associated with the Respondent Department, any of its committees, or the examination preparation process. He oversees actual reportage of dangerous chemical spills for the Department of Community Affairs. Although Mr. Dawkins indicated that he, personally, would not have written question 78 quite the way it was posed on the examination, he still felt that since it addressed reporting requirements, examinees should have assumed that a reportable quantity was involved and answered accordingly. All three of Respondent's experts testified that the answer chosen as correct by the Department was the most accurate of the multiple choice answers provided on the examination and that the subject matter and correct answer should have been understood by a qualified operator of a wastewater treatment plant at the "B" licensure level. The Department has under contract an expert in examination and psychometrics. The Evaluation Services Instructional Support Center Learning Systems Institute of Florida State University provides to the Department as part of the examination grading, a special analysis report for each examination. This report contains statistics about the scores, difficulty of each question, and how the spread of answers by the examinees compared to the four quadrants of grade results. The February 1998 examination was an extremely difficult examination, as evidenced by the fact that more examinees failed than passed. However, it was demonstrated that 77% of examinees who took the examination got question 78 correct. Question 78 also discriminated between high and low scoring examinees. The item analysis performed before the other four questions were deleted did not show that question 78 was misleading in any way, but did show that each of the four questions deleted were misleading or otherwise flawed. One of the proctors for the February 1998 examination personally observed that at the time the examination ended, only two examinees remained in the examination room and that neither of these examinees was Petitioner. It can be inferred therefrom that Petitioner had finished the examination, had time to spare, and had left the room. Finally, the inclusion of examination questions which were later deleted is not a concern as to the time allotted. This type of examination is a "power exam" and speed is not a factor.
Recommendation Upon the foregoing findings of fact and conclusions of law, it is RECOMMENDED that the Department of Environmental Protection enter a Final Order denying the Petitioner's challenge to the February 1998 Class "B" wastewater operator certification examination and assigning her a final percentage grade of 63% thereon. DONE AND ENTERED this 16th day of November, 1998, in Tallahassee, Leon County, Florida. ELLA JANE P. DAVIS Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 SUNCOM 278-9675 Fax Filing (850) 921-6847 Filed with the Clerk of the Division of Administrative Hearings this 16th day of November, 1998. COPIES FURNISHED: Gerrit Vanthul, Qualified Representative 5279 Southeast 39th Street Trenton, Florida 32693 Cynthia Christen, Esquire Department of Environmental Protection 2600 Blairstone Road Mail Station 35 Tallahassee, Florida 32399-3000 Kathy Carter, Agency Clerk Department of Environmental Protection 3900 Commonwealth Boulevard Mail Station 35 Tallahassee, Florida 32399-3000 F. Perry Odom, General Counsel Department of Environmental Protection 3900 Commonwealth Boulevard Mail Station 35 Tallahassee, Florida 32399-3000
The Issue Did Petitioner pass the Mechanical Engineers Examination he took on April 24, 1998?
Findings Of Fact On April 24, 1998, Petitioner took the Mechanical Engineers Examination. He received a score of 69 for his effort. A passing score was 70. The Mechanical Engineers Examination was administered under Respondent's auspices. As alluded to in the preliminary statement, Petitioner challenged the score received on problem 146. The maximum score available for that problem was ten points. Petitioner received eight points. In accordance with the National Council of Examiners for Engineering and Surveying Principles in Practice of Engineering Examinations for spring 1998, score conversion table - discipline specific, Petitioner had a raw score of 47 which equated to a conversion of 69, to include the eight raw points received for problem 146. In addition, the examination provided a scoring plan for problem 146, which assigns scores in increments of two points from zero to ten. To pass, it would be necessary for Petitioner to receive an incremental increase of two points, raising his score from eight points to ten points. This would give him a raw score of 49 points. According to the score conversion table - discipline specific, that would give Petitioner 71 points. According to the scoring plan for problem 146 to receive the ten points, Petitioner would have to demonstrate: Exceptional competence (it is not necessary that the solution to the problem be perfect) generally complete, one math error. Shows in-depth understanding of cooling load calculation psychrometrics. Problem 146 required Petitioner to: Determine the required cooling coil supply air quantity (cfm) and the conditions (°F db and °F wb) of the air entering and leaving the coil." Petitioner was provided a psychrometric chart to assist in solving problem 146. The examination candidates were also allowed to bring reference sources to the examination to assist in solving the examination problems. Petitioner brought to the examination, the Air-Conditioning Systems Design Manual prepared by the ASHRAE 581-RP Project Team, Harold G. Lorsch, Principal Investigator. Petitioner used that manual to determine the wet-bulb temperature of the air entering the coil. In particular, he used an equation from the manual involving air mixtures. For that part of the solution he arrived at a temperature of 65.6°F wb. According to the problem solution by Respondent's affiliate testing agency, reference ASHRAE Fundamentals Chapter 26, the coil entering wet-bulb temperature taken from the psychrometric chart was 66.12°F wb. The scorer in grading Petitioner's solution for problem 146 placed an "x" by the answer provided 65.6°F wb and wrote the words "psychrometric chart." No other entry or comment was made by that scorer in initially reviewing the solution Petitioner provided for that problem. This led to the score of eight. The scoring plan for problem 146 for the April 1998 examination taken by Respondent equates the score of eight as: MORE THAN MINIMUM BUT LESS THAN EXCEPTIONAL COMPETENCE Either a) Provides correct solution to problem with two math errors or incorrect dry-bulb or wet-bulb for coil entering or leaving conditions or minor total cooling load error, or b) Provides correct solution to items c and d correctly and minor math errors in items a and b of Score 6 below. Petitioner was entitled to review the results of his examination. He exercised that opportunity on September 21, 1998, through a post-examination review session. Petitioner requested and was provided re-scoring of his solution to problem 146. According to correspondence from the National Council of Examiners for Engineering and Surveying to the Florida Member Board from Patricia M. Simpson, Assistant Supervisor of scoring services, the score did not change through re-scoring. In this instance, the October 14, 1998 correspondence on re-scoring states, in relation to problem 146: Incorrect methodology used in calculating coil entering wet-bulb temperature. Incorrect coil entering wet-bulb temperature provided. No calculation provided for coil leaving temperature conditions. The coil leaving wet-bulb temperature in Respondent's proposed solution was 53.22°F wb taken from the psychrometric chart. Petitioner's solution for the coil leaving wet-bulb temperature taken from the psychrometric chart was 53.3°F wb. At hearing Respondent did not provide an expert to establish the basis for point deduction in the original score and the re-scoring of Petitioner's solution for problem 146. Moreover, Respondent did not present expert witnesses to defend the commentary, the preferred written solution in its examination materials. Consequently, Respondent's preferred solution constitutes hearsay about which no facts may be found accepting the validity of Respondent's proposed solution, as opposed to merely reporting that information.1 By contrast, Petitioner provided direct evidence concerning the solution provided for problem 146 in response to the criticisms of his solution that were unsupported by competent evidence at hearing. More importantly the criticisms were responded to at hearing by Geoffrey Spencer, P.E., a mechanical engineer licensed to practice in Florida, who was accepted as an expert in that field for purposes of the hearing. As Petitioner explained at hearing, he used the Air- Conditioning Systems Design Manual equation to arrive at the coil entering wet-bulb temperature, which he believed would provide the answer as readily as the use of the psychrometric chart. (Although the psychrometric chart had been provided to Petitioner for solving problem 146, the instructions for that problem did not prohibit the use of the equation or formula.) Petitioner in his testimony pointed out the equivalency of the process of the use of the psychrometric chart and the equation. Petitioner deemed the equation to be more accurate than the psychrometric chart. Petitioner had a concern that if the answer on the coil entering wet-bulb temperature was inaccurate, this would present difficulty in solving the rest of problem 146 because the error would be carried forward. Petitioner pointed out in his testimony that the solution for determining the coil entering wet-bulb temperature was set out in his answer. The answer that was derived by use of the formula was more time consuming but less prone to error, according the Petitioner's testimony. Petitioner points out in his testimony that the answer he derived, 65.6°F wb, is not significantly different than Respondent's proposed solution of 66.12°F wb. (The instructions concerning problem 146 did not explain what decimal point of a degree the candidate had to respond to in order to get full credit for that portion of the solution to the problem.) Petitioner in his testimony concerning his solution for the coil leaving wet-bulb temperature indicated that the calculation for arriving at that temperature was taken from the psychrometric chart and is sufficiently detailed to be understood. Further, Petitioner testified that the degree of accuracy in which the answer was given as 53.3°F wb, as opposed to Respondent's proposed solution of 53.22°F wb, is in recognition of the use of the psychrometric chart. Petitioner questions whether the proposed solution by Respondent, two decimal points, could be arrived at by the use of the psychrometric chart. In relation to the calculation of the coil entering wet-bulb temperature, Mr. Spencer testified that the formula from the Air-Conditioning Systems Design Manual or the psychrometric chart could have been used. Moreover, Mr. Spencer stated his opinion that the solution for coil entering wet-bulb temperature of 65.6°F wb by Petitioner is sufficiently close to Respondent's proposed solution of 66.12°F wb to be acceptable. Mr. Spencer expressed the opinion that Petitioner had correctly used the formula from the manual in solving the coil entering wet-bulb temperature. Mr. Spencer expressed the opinion that the psychrometric chart is an easier source for obtaining the solution than the use of the formula from the manual. In Mr. Spencer's opinion, the formula shows a more basic knowledge of the physics involved than the use of the psychrometric chart would demonstrate. In relation to the coil leaving wet-bulb temperature, Mr. Spencer expressed the opinion that Petitioner had adequately explained the manner of deriving the answer. Further, Mr. Spencer expressed the opinion that the answer derived was sufficiently accurate. The testimony of Petitioner and opinion of Mr. Spencer is unrefuted and accepted.
Recommendation Upon consideration of the facts found and conclusions of law reached, it is RECOMMENDED: That a final order be entered which finds that Petitioner passed the Florida Board of Professional Engineers April 24, 1998, Mechanical Engineers Examination with a score of 71. DONE AND ENTERED this 22nd day of February, 1999, in Tallahassee, Leon County, Florida. CHARLES C. ADAMS Administrative Law Judge Division of Administrative Hearings The DeSoto Building 1230 Apalachee Parkway Tallahassee, Florida 32399-3060 (850) 488-9675 SUNCOM 278-9675 Fax Filing (850) 921-6847 www.doah.state.fl.us Filed with the Clerk of the Division of Administrative Hearings this 22nd day of February, 1999.
