STATE OF FLORIDA

DIVISION OF ADMINISTRATIVE HEARINGS

CLASINA VANTHUL, )

)

Petitioner, )

)

vs. ) Case No. 98-2429

) DEPARTMENT OF ENVIRONMENTAL ) PROTECTION, )

)

Respondent. )

)

RECOMMENDED ORDER

Upon due notice, this cause came on for formal hearing on September 11, 1998, in Tallahassee, Florida, before Ella Jane P. Davis, a duly assigned Administrative Law Judge of the Division of Administrative Hearings.

APPEARANCES

For Petitioner: Gerrit Vanthul, as Qualified Representative 5279 Southeast 39th Street

Trenton, Florida 32693

For Respondent: Cynthia Christen, Esquire

Department of Environmental Protection 3900 Commonwealth Boulevard

Mail Station 35

Tallahassee, Florida 32399-3000 STATEMENT OF THE ISSUES

1. Is Petitioner entitled to credit for her answers to any of the questions she specifically challenged or for the four questions deleted by the Department of Environmental Protection (Department) on the February 1998 Class "B" Domestic Wastewater Operator Certification examination (wastewater examination)?

2. Was the Department's administration or grading of Petitioner's examination arbitrary, capricious or otherwise unfair so as to entitle Petitioner to either additional points for a passing grade or an opportunity to retake the examination without cost?

PRELIMINARY STATEMENT

By letter of April 6, 1998, the Department notified Petitioner that she had failed the February 1998 wastewater examination. Petitioner timely requested an administrative hearing by a letter dated May 5, 1998. The Department referred the matter to the Division of Administrative Hearings on or about June 6, 1998. At the time of its referral, the Department had already received another letter from Petitioner dated May 26, 1998. The second letter alleged additional disputed issues of material fact. The Department never forwarded the May 26, 1998, letter to the Division.

On June 30, 1998, an Order of Prehearing Instructions and Notice of Hearing for September 11, 1998, was mailed to both parties.

At the commencement of the disputed fact hearing, the Department stipulated that Petitioner's second letter could be treated as an amended petition and that the Department had adequate notice of the additional disputed issues of material fact included in Petitioner's second letter. Accordingly, the case went forward on the merits of all issues raised.

After appropriate examination on the record, and without objection by the Department, Gerrit Vanthul, Petitioner's husband, was accepted as Petitioner's Qualified Representative for purposes of this proceeding.

Respondent Department had timely complied with the Order of Prehearing Instructions. Petitioner had not complied with the Order of Prehearing Instructions. Nonetheless, Petitioner was permitted to present documentary evidence and her sole witness.

Petitioner presented the oral testimony of Gerrit Vanthul and had eight exhibits admitted in evidence. Part of Petitioner's Exhibit 1 was not admitted, and Petitioner's Exhibit

9 was not admitted. Petitioner did not testify on her own behalf.

Respondent Department presented the oral testimony of Greg Dawkins, Mary Smith, Mary Patt Peterson, Gary Peterson, Ph.D., and William Allman. Greg Dawkins was accepted as an expert in emergency response and community right-to-know. Dr. Gary Peterson was accepted as an expert in examination development and psychometrics. William Allman was accepted as an expert in the operation of wastewater treatment plant operations. Respondent had nine exhibits admitted in evidence.

A transcript was filed in due course, and all timely filed proposed recommended orders have been considered in the preparation of this recommended order.

FINDINGS OF FACT

1. Petitioner took the Class "B" wastewater operator certification examination given in February 1998 by the Department of Environmental Protection.

2. Pilot questions are questions submitted by licensees and educators which do not yet have a "performance record" of testing validity. Petitioner alleged that pilot questions were used on her examination contrary to rules of the Department. Petitioner submitted no competent evidence to establish this allegation and it was credibly refuted. There are no pilot questions in the bank of potential questions from which the examination was composed.

3. Petitioner conjectured that questions on her "B" level examination may have been drawn from a bank of questions for a higher level ("A") certification examination. In fact, the examination questions for the "B" level examination were selected from a bank of questions developed by the Department of Business and Professional Regulation. The Department of Business and Professional Regulation was the agency that had jurisdiction over the operator certification program before the Department of Environmental Protection assumed responsibility therefor. The selection of the examination questions was accomplished by selecting the percentage of questions from a range in a subject area already predetermined by rule and a computer program inserting the number of questions to fill that percentage. There is no way the computer program can select questions from another

   level of examination, for instance "A" level or "C" level.

   
   

4. Prior to the examination, candidates for examination were advised they would have four hours to complete the examination. Examinees for the February 1998 examination in fact were provided four full hours after all preliminary matters and reading of instructions were completed.

5. Prior to the examination, the Department provided candidates for examination with a list of subject areas that it intended to place on the examination, so that candidates could effectively prepare. All subject areas, except one, were in fact covered on the February 1998 examination. In some instances, a single question/answer satisfied two or more subject areas because of content equally applicable to each subject area. In other instances, the same subject area was covered by several questions/answers. Only one subject area that was listed in the pre-examination information did not appear on the February 1998 examination. That subject area was "energy." The reason that the subject area of "energy" was not included on the February 1998 examination was that there were no energy questions in the bank of questions which the Department of Environmental Protection had inherited from the Department of Business and Professional Regulation.

6. The sole result of the absence of any energy question on the examination is that Petitioner and all other examinees in her group did not receive as thorough an examination in a single

   subject matter area as the licensure board had aspirationally intended. However, all examinees were equally treated.

7. Originally, there were 87 questions on the February 1998 examination. After the examination was administered and a special analysis report on the grades was produced, the Department's examination review committee met with the examination consultants. The committee recommended to the Department, and the Department accepted the recommendation, that four questions should be deleted because they were misleading or for some other reason failed to adequately and reliably measure the examinees' ability to practice at a Class "B" license level. Removal of the four questions only lowered the mean score by one point, thereby creating a slightly easier examination while simultaneously slightly increasing its reliability.

8. Examinees were instructed to select the best multiple choice answer for each question. Each of the questions was equally weighted.

9. The Department interpreted Rule 61E12-41.005(5), Florida Administrative Code, as requiring that examinees achieve at least a 65% rounded score on the examination in order to pass the examination.

10. In order to determine an examinee's success on the examination, the Department multiplied the initial 87 questions by 65% (.65) and so determined that an examinee would need at least 54 correct questions/answers to earn a passing grade. In

    determining a candidate's grade on an operator licensure examination, the Department determines the number of correct answers needed to reach the minimum rounded score of 65%. A special analysis report also indicates how many correct answers equal each percentage grade. If this number is not a whole number, the Department uses the rounding method to reach a whole number, based on 0.5 +/- percentage.

11. By the foregoing grading interpretation, before deletion of the four questions, Petitioner's rounded score was 60%, with 52 correct answers. Petitioner's grade improved with the deletion of the four questions, because she had incorrectly answered each of the four questions which were later deleted. After the four questions were deleted, the same grading system resulted in a rounded score of 63% with 52 correct answers.

12. By letter dated April 6, 1998, the Department notified Petitioner that she had failed the examination because she did not get a rounded 65% score based on 52 correct answers.

13. After receiving the letter, Petitioner requested a review of the examination. Petitioner was allowed to review the questions and answers she had missed. Petitioner was also allowed to write comments on the question sheets which she reviewed.

14. Petitioner's comments were submitted to the examination review committee of the Department for the committee's review. Upon review of Petitioner's comments and the examination, the

    committee determined that the questions and required answers were accurate and fair. It recommended no change to Petitioner's score. Petitioner was notified that no change would be made to her score.

15. Petitioner then timely requested an administrative hearing. Although Petitioner's two letters/petitions (see Preliminary Statement supra.) initially raised issues concerning a number of examination questions, Petitioner only presented evidence concerning the contents of question 78 at formal hearing.

16. Question 78 tested examinees' knowledge of appropriate emergency response activity and notification concerning the release of chlorine gas.

    
    

17. Petitioner asserted that question 78 was vague, ambiguous, and misleading because it did not specifically state that a "reportable quantity" was to be considered in choosing the best answer from among multiple choice options of reporting a chlorine spill to one entity, two entities, three entities or no entities. For this reason, Petitioner alleged that her answer could have been an answer which was equally correct ("multi- keyed") with the answer selected as correct by the Department.

18. Mr. Dawkins, who was accepted as an expert in emergency response and community right-to-know, testified that the question was not misleading. Mr. Dawkins is not associated with the

    Respondent Department, any of its committees, or the examination preparation process. He oversees actual reportage of dangerous chemical spills for the Department of Community Affairs.

    Although Mr. Dawkins indicated that he, personally, would not have written question 78 quite the way it was posed on the examination, he still felt that since it addressed reporting requirements, examinees should have assumed that a reportable quantity was involved and answered accordingly.

19. All three of Respondent's experts testified that the answer chosen as correct by the Department was the most accurate of the multiple choice answers provided on the examination and that the subject matter and correct answer should have been understood by a qualified operator of a wastewater treatment plant at the "B" licensure level.

20. The Department has under contract an expert in examination and psychometrics. The Evaluation Services Instructional Support Center Learning Systems Institute of Florida State University provides to the Department as part of the examination grading, a special analysis report for each examination. This report contains statistics about the scores, difficulty of each question, and how the spread of answers by the examinees compared to the four quadrants of grade results.

21. The February 1998 examination was an extremely difficult examination, as evidenced by the fact that more examinees failed than passed. However, it was demonstrated that

    77% of examinees who took the examination got question 78 correct. Question 78 also discriminated between high and low scoring examinees.

22. The item analysis performed before the other four questions were deleted did not show that question 78 was misleading in any way, but did show that each of the four questions deleted were misleading or otherwise flawed.

23. One of the proctors for the February 1998 examination personally observed that at the time the examination ended, only two examinees remained in the examination room and that neither of these examinees was Petitioner. It can be inferred therefrom that Petitioner had finished the examination, had time to spare, and had left the room.

24. Finally, the inclusion of examination questions which were later deleted is not a concern as to the time allotted. This type of examination is a "power exam" and speed is not a factor.

    
    

    CONCLUSIONS OF LAW

    
    

25. The Division of Administrative Hearings has jurisdiction over the parties and subject matter of this cause, pursuant to Section 120.57(1), Florida Statutes.

26. The burden of proof and duty to go forward is upon the Petitioner in this cause. Petitioner must show by a preponderance of the evidence that the examination was faulty,

    arbitrarily or capriciously worded or graded, or that Petitioner was arbitrarily or capriciously denied credit through a grading process devoid of logic or reason. Harac v. Department of Professional Regulation, 484 So. 2d 1333, 1338 (Fla. 3rd DCA 1986); State ex rel Glaser v. J.M. Pepper, 155 So. 2d 383 (Fla. 1st DCA 1963).

27. For the regulation of operators, the Department of Environmental Protection is entitled to use rules developed by the Department of Business and Professional Regulation until the Department of Environmental Protection promulgates its own rules, as set forth in Section 17 of Chapter 97-236, Laws of Florida.

28. Rule 61-11.010(1)(b), Florida Administrative Code, provides as follows:

    1. Pursuant to Section 455.217, Florida Statutes, grading of all examinations shall be processed only as follows:

       
       

       * * *

       (b) Departmentally developed objective, multiple choice examination shall be graded by the Department or its designee. After an examination has been administered the Board shall reject any questions that do not reliably measure the general areas of competency specified in the rules of the Board. The Department shall review the item analysis and any statistically questionable items after the examination has been administered. Based upon this review, the Department shall adjust the scoring key by totally disregarding the questionable items for grading purposes or by multi-keying, giving credit for more than one correct answer per question. All questions which do not adequately and reliably measure the applicant's ability to practice the profession shall be rejected. The Department

       shall calculate each candidate's grade utilizing the scoring key or adjusted scoring key, if applicable, and shall provide each candidate a grade report.

       
       

29. Therefore, Petitioner had no right to have the four deleted questions scored as part of her examination.

30. Petitioner failed to establish that she suffered any disadvantage as a result of having too little time to complete the examination. Her theory that she would have achieved more correct answers and thus a higher percentage score if she had spent less time or no time on the four questions which were ultimately deleted and more time on the remaining 83 questions which ultimately were counted, is purely speculative and is belied entirely by the fact that she left the examination room before the entire time allotted for taking the examination had elapsed.

31. Petitioner's theory that she would have done better on other subject matter areas of the examination if she had not spent time studying the subject matter of "energy" which was not included in the examination also is pure speculation. Her assertion that she was somehow prejudiced by not being examined at all in the subject area of energy also is purely speculative and does not indicate any arbitrary, capricious, or prejudicial treatment by the Department under the facts of this case.

32. Petitioner's theory with regard to the correct answer on question 78 was not proven, and clear, competent, and substantial evidence was presented by Respondent to demonstrate

    that question 78 was not a pilot question or a "trick" question, or otherwise arbitrary or capricious. Moreover, clear and convincing evidence (not just preponderant evidence) was produced by the Department to show that Petitioner did not select the best multiple choice answer for question 78. Question 78 was incorrectly answered by Petitioner. Petitioner's answer to question 78 was not "multi-keyed" because to do so would have given her credit for a wrong answer The people of Florida would be best protected by licensing at the Class "B" level only persons with the required skills, education, training, and experience necessary to be able to correctly answer question 78.

33. Petitioner's proposed calculation of her score as set out in various ways in the testimony of Mr. Vanthul, several exhibits, and Petitioner's proposed recommended order, is not in accord with the clear requirements of Rule 61E12-41.005(5), Florida Administrative Code.

34. The Department's methodology of scoring as set forth in the Findings of Fact, supra, also does not follow the precise language of the rule. The Department applied the percentage backwards to determine the number of correct questions/answers each examinee would need to equal 65%, instead of dividing the number of possible correct answers into each respective examinee's number of actual correct answers so as to determine each examinee's exact correct percentage score and then rounding. However, this makes no difference because the Department got the

    correct percentages.

    
    

35. Even when the clear language of the rule is literally applied, Petitioner has failed to make a minimum passing score of 65%.

36. Rule 61E12-41.005(5) reads:

Examination answer sheets shall be electronically scored. The minimum passing score on the examination is 65%. In rounding percentages, any percentage which is 0.5 or above shall be rounded up to the next higher whole number. Percentages less than 0.5 shall be rounded to the next lower whole number.

1. If the clear language of the rule is literally applied, the following calculations are made: Originally, there were 87 equally-weighted questions on the examination, and Petitioner got

   52 correct answers. 52 correct answers out of 87 possible correct answers constitutes an exact percentage score of 59.77%. Using the "rounding" rule as written, this would have rendered Petitioner's score as 60%. Petitioner had incorrectly answered the four deleted questions, so their removal from consideration gave her 52 correct answers out of 83 possible correct questions.

   52 out of 83 constitutes an exact percentage score of 62.65%, which rendered Petitioner's score, according to a literal application of the "rounding" rule, as 63%, or two percent short of a minimum passing rounded score of 65% on the examination.

2. The examination was not faulty. All candidates' grades were calculated in the same manner. No one was given credit for any of the four deleted questions. Neither the wording nor the

grading of those questions were arbitrary or capricious. The Department's refusal to credit Petitioner for correct answers on the four deleted questions or for the one specifically challenged question 78 was not devoid of logic or reason.

RECOMMENDATION

Upon the foregoing findings of fact and conclusions of law, it is

RECOMMENDED that the Department of Environmental Protection enter a Final Order denying the Petitioner's challenge to the February 1998 Class "B" wastewater operator certification examination and assigning her a final percentage grade of 63% thereon.

DONE AND ENTERED this 16th day of November, 1998, in Tallahassee, Leon County, Florida.

ELLA JANE P. DAVIS

Administrative Law Judge

Division of Administrative Hearings The DeSoto Building

1230 Apalachee Parkway

Tallahassee, Florida 32399-3060

(850) 488-9675 SUNCOM 278-9675

Fax Filing (850) 921-6847

Filed with the Clerk of the Division of Administrative Hearings this 16th day of November, 1998.

COPIES FURNISHED:

Gerrit Vanthul, Qualified Representative 5279 Southeast 39th Street

Trenton, Florida 32693

Cynthia Christen, Esquire Department of Environmental

Protection

2600 Blairstone Road

Mail Station 35

Tallahassee, Florida 32399-3000

Kathy Carter, Agency Clerk Department of Environmental

Protection

3900 Commonwealth Boulevard

Mail Station 35

Tallahassee, Florida 32399-3000

F. Perry Odom, General Counsel Department of Environmental

Protection

3900 Commonwealth Boulevard

Mail Station 35

Tallahassee, Florida 32399-3000

NOTICE OF RIGHT TO SUBMIT EXCEPTIONS

All parties have the right to submit written exceptions within 15 days from the date of this Recommended Order. Any exceptions to this Recommended Order should be filed with the agency that will issue the final order in this case.

Docket for Case No: 98-002429

Orders for Case No: 98-002429