STATE OF FLORIDA

DIVISION OF ADMINISTRATIVE HEARINGS

DON BATES, )

)

Petitioner, )

)

vs. ) CASE NO. 90-2164

) BETTY CASTOR as COMMISSIONER OF ) EDUCATION, )

)

Respondent. )

)

RECOMMENDED ORDER

Pursuant to notice, the Division of Administrative Hearings, by its duly designated Hearing Officer, Claude B. Arrington, held a formal hearing in the above-styled case on April 22 and 23, 1991, in Tallahassee, Florida.

APPEARANCES

For Petitioner: Thomas W. Young, III, Esquire

FEA/United

118 North Monroe Street Tallahassee, Florida 32399-1700

For Respondent: Charles S. Ruberg, Esquire

Assistant General Counsel Florida Board of Education The Capitol, Suite 1701

Tallahassee, Florida 32399-0400 STATEMENT OF THE ISSUE

Whether the scores given to Petitioner's essay on the communication subtest of the May 1989 Florida Education Leadership Examination were inconsistent with and the result of an improper application of the holistic scoring methodology, and whether the scores given to his essay on the communication subtest of the November 1989 Florida Education Leadership Examination were inconsistent with and the result of an improper application of the holistic scoring methodology.

PRELIMINARY STATEMENT

Petitioner, a professional educator, sat for the Florida Education Leadership Examination (FELE) in May 1989. After Petitioner was advised that he had not passed the communications subtest of the May 1989 FELE, he attempted to determine the reasons he did not pass. While he was awaiting a response from Respondent as to the May 1989 scores, he retook the communications subtest in November 1989. After he was advised that he had not passed the communication subtest in either May or November 1989, he filed these challenges to the manner in which the essay portion on both the May 1989 FELE and the November 1989 FELE were scored. Petitioner contends that a proper application of the holistic

scoring method, the method set forth by rule, would produce a higher score. Petitioner does not challenge the underlying rule that requires application of the holistic scoring method.

At the formal hearing, Petitioner testified on his own behalf and presented the additional testimony of Betty Owen, Charles R. Blackmon, and David Kirby.

Ms. Owen is an experienced reader for the FELE and was one of the readers who scored Petitioner's essay for the November 1989 FELE. Dr. Blackmon is a retired educator and educational administrator who was accepted as an expert witness in the field of educational administration and in the evaluating written communication using the holistic scoring method. Dr. Kirby is a McKenzie professor of English at Florida State University and was accepted as an expert witness in the field of evaluating written communication using the holistic scoring method. The parties submitted the following joint exhibits which were accepted into evidence: A1-6, B1-6, C1-4, D, E, F, and G. Petitioner submitted one additional exhibit, which was accepted into evidence. Respondent submitted three additional exhibits, which were accepted into evidence.

A transcript of the proceedings has been filed. At the request of the parties, the time for filing post-hearing submissions was set for more than ten days following the filing of the transcript. Consequently, the parties waived the requirement that a recommended order be rendered within thirty days after the transcript is filed. Rule 22I-6.031, Florida Administrative Code. Rulings on the parties' proposed findings of fact may be found in the Appendix to this Recommended Order.

FINDINGS OF FACT

1. Petitioner is a professional educator who moved to Florida in 1988 to teach in the Broward County public school system. He completed Florida's beginning teacher program and passed the Florida Teacher Certification Examination. At the time of the formal hearing, Petitioner was employed as a social studies teacher in one of the public schools of Broward County.

   
   

2. Petitioner received his Bachelor of Science degree from East Tennessee State University (ETSU) in 1967, majoring in Political Science and Speech. In 1971, he received a Masters of Art degree in Educational Administration and Supervision with a minor in Instructional Communication from ETSU. He earned his Doctorate of Education degree from the University of Alabama in 1974, with a major in Educational Administration and Supervision and a minor in Curriculum Development and Instructional Communication.

   
   

3. Petitioner has been teaching continuously since 1967, either at the high school or college level. He has considerable experience in reading and evaluating written communication of high school and graduate students. He has served as an assistant principal in the State of Alabama and holds superintendent's credentials as well as principal's credentials and supervisory and teaching credentials for the states of Alabama and Louisiana. Petitioners desires to work as an assistant principal in the public schools of Florida.

   
   

4. A person who desires to hold the position as a principal or as an assistant principal in a Florida public school district must earn an Educational Leadership Certificate from Respondent. Among the requirements for achieving this Certificate is the successful completion of the Florida Education Leadership Examination (FELE).

5. FELE is administered twice each year to those seeking certification in educational leadership. The examination covers the eight areas of the Florida Educational Leadership Core Curriculum listed in Rule 6A-4.0082, Florida Administrative Code. The candidates for certification are notified by the FELE registration bulletin that the examination consists of three subtests with Subtest One covering "Management, Leadership, Personnel"; Subtest Two covering "Communications (includes essay)"; and Subtest Three covering "Curriculum, Finance, Law, Technology". The registration bulletin also notifies candidates that:

   
   

   With the exception of the Communications essay question, all questions will be multiple choice. There will be one essay question designed to demonstrate proficiency in written communication.

   The essay question will be scored holistically by trained experts.

   
   

6. As indicated by the registration bulletin, the Communications Subtest consists of multiple choice questions and of an essay. The scores received on the two parts of the Communications Subtest are combined and statistically adjusted to determine whether the candidate passed the subtest. (There are several different versions of FELE that have been developed over the years. The statistical adjustment is made to maintain consistency in the examination process by making adjustments in the scoring process to account for differences in the comparative difficulty between the different versions of FELE. No challenge is raised to the fact that statistical adjustments are made or as to how those adjustments were made for the examinations in question.)

   
   

7. A candidate must pass all three subtests of the FELE in order to receive certification, and he can retake any subtest that he failed to pass.

   
   

8. A candidate chooses the topic for his essay from the two topics that are offered by the essay portion of the Communications Subtest. While the topics relate to assignments that a school administrator might receive, the essay portion of the examination is designed to test the candidate's communication skills; it is not designed to test the candidate's administrative skills or substantive knowledge of the topic of the essay.

   
   

9. Rule 6A-4.00821, Florida Administrative Code, pertains to FELE and provides, in pertinent part, as follows:

   
   

   1. Scoring.

      (b) The essay portion to the school communications subtest shall be scored by two (2) trained judges using a scale of one (1) which is an unsatisfactory score to four (4) which is an outstanding score.

      In the event the two (2) ratings are two (2) or more points different, or in the event the summed ratings equal three (3), the writing sample will be rated by a referee and the referee's score will replace the most discrepant of the original ratings.

      (d) Beginning July 1, 1988, a passing score for each subtest of the Florida Education Leadership Examination shall be:

      1. School Communications. Examinee scores for the school communications subtest shall be reported as an average scaled score from the essay test and the scaled score from the multiple choice questions. The passing score shall be the scaled score equivalent to the combination of the essay total raw score of four (4) and a multiple choice total raw score of fifteen (15) on the November, 1987 administration of the subtest.

   2. Essay performance standards.

      (c) Rating scale. The four-level scale for judging the written essays is defined as follows:

      1. A rating of one (1) indicates the essay lacks unity and focus. It is distorted or ambiguous, and it fails to treat the topic in sufficient depth and breadth. There is little or no discernible organization and only scant development of ideas, if any at all. The essay betrays only sporadically a sense of paragraph and sentence structure, and it is syntactically slipshod. Usage is irregular and

         often questionable or wrong. There are serious errors in spelling, capitalization, and punctuation.

      2. A rating of two (2) indicates the essay has some degree of unity and focus, but each could be improved. It is reasonably clear, though not invariably so, and it treats the topic with a marginal degree of sufficiency. The essay reflects some concern for organization and for some development of ideas, but neither is necessarily consistent nor fully realized. The essay reveals some sense, if not full command of paragraph and sentence structure. It is syntactically bland and, at times, awkward. Usage is generally accurate, if not consistently so. There are some errors in spelling, capitalization, and punctuation that detract from the essay's effect if not from its sense.

      3. A rating of three (3) indicates the essay is focused and unified, and it is clearly if not distinctly written. It gives the topic an adequate though not always thorough treatment. The essay is well organized, and much of the time it develops ideas appropriately and sufficiently. It shows a good grasp of paragraph and sentence structure, and its usage is generally accurate and sensible. Syntactically, it is clear and reliable. There may be a few errors in spelling, capitalization, and punctuation, but they are not serious.

      4. A rating of four (4) indicates the essay is unified, sharply focused, and distinctively effective. It treats the topic clearly, completely, and in suitable depth and breadth. It is clearly and fully organized, and it develops ideas with consistent appropriateness and thoroughness. The essay reveals an unquestionably firm command of paragraph and sentence structure. Syntactically, it is smooth and often elegant. Usage is uniformly sensible, accurate, and sure. There are very few, if any errors in spelling, capitalization, and punctuation.

10. In addition to the foregoing, Rule 6A-4.00821, Florida Administrative Code, contains minimal qualifications for those persons who will score the essays. The chief reader and the scorers involved in the scoring of Petitioner's essays for both the May 1989 and the November 1989 examinations are well qualified, appropriately trained, and experienced in the application of the holistic scoring method. 1/

    
    

11. The holistic scoring method is widely used to evaluate essays such as the ones on the FELEs involved in this case. 2/ Unlike the analytical method of evaluating essays, the holistic method is designed only for scoring purposes and is not designed to give the writer feedback as to how he can improve his writing or why the essay received a particular score. For that reason, Petitioner received no feedback following the examinations, and he was not informed what he needed to do to improve his essay.

    
    

12. The following instructions were among the general instructions given to the candidates for the May 1989 FELE and the November 1989 FELE:

    
    

    Your essay will be judged on your ability to write in a logical, easily understood style and on the quality of information provided, NOT on the opinion expressed.

    
    

13. The following essay rating criteria were provided the candidates for the May 1989 FELE and the November 1989 FELE:

    
    

    Unity, organization, and focus (described as "consistency, coherence, order and interdependence of parts, creating a single, integrated effect")

    
    

    Sufficiency and development (described as "appropriateness of expression to meet writer's and subject's needs; breadth and depth; use of detail, examples, illustrations")

    
    

    Usage and syntax (described as "appropriate ordering of words to convey intended meaning; appropriate use of language, inflection, tense, agreement, vocabulary")

    
    

    Clarity and paragraph and sentence structure (described as "lucidity of expression; paragraph structure; variety, logic, and relatedness of sentences within paragraphs")

    
    

    Spelling, capitalization, punctuation (described as "currently accepted standards")

    
    

14. To produce reliable and consistent scoring of the essays, a well developed process is closely followed. The first stage of the holistic scoring process is the selection of a chief reader, an assistant chief reader, and readers. 3/

15. The chief reader meets with the assistant chief reader and with other readers, normally selected because of their experience, to select range finders and sample papers. This group, referred to as selectors, first discusses the scoring definitions and builds a consensus as to the criteria by which the papers will be judged. The members of the group review range finders from a previous administration of the examination so that the criteria used will be consistent with past scoring. They then read, rank, and discuss 40-50 sample papers, selected at random from the current examination. Thereafter, in a process that builds towards a consensus as to what scores the different sample papers should receive, they discuss what ratings these papers received and how the rating criteria applies. They then select at least six range finders as being typical of the one, two, three, and four categories plus two scores that are borderline. These range finders are backed up by 15-20 sample papers which also exemplify scale level and borderline cases. It is the responsibility of the chief reader to determine that the selectors have reached an appropriate understanding as to the criteria to be applied to scoring these essays.

    
    

16. After the range finders and the samples have been selected, the chief reader and the assistant chief readers meet with those readers who will have the responsibility of scoring the essays. The process used to build consensus among the selectors as to the appropriate scoring criteria is followed to build consensus among the readers. It is the responsibility of the chief reader to determine that all readers have a clear understanding as to the criteria to be used in scoring the essays. The actual scoring does not begin until after the chief reader has made that determination. Throughout the scoring process, the chief reader (or the table leader, if an examination is taken by enough people to justify a table reader) monitors the reading to ensure that the readers are applying the appropriate criteria in scoring the essays.

    
    

17. Each essay is independently scored by two readers, each of whom is unaware of the score given by the other reader. No participant in the scoring process knows the identity of the essay writer.

    
    

18. Essays which receive the same or contiguous scores from two readers are completed and those scores are recorded. For example, if the two readers each scored an essay as a two, the total score the candidate would receive on his essay would be a four. If one reader scored the essay as a two while the other scored it as a three, the total score the candidate would receive on his essay would be a five. That score would be added to the score he received on the multiple choice portion of the communications subtest and statistically adjusted to determine his final score on the communications subtest. The passing score for the communications subtest is determined as provided by Rule 6A-4.00821(7)(d)1, Florida Administrative Code.

    
    

19. Petitioner took the FELE in May 1989. He failed the communications subtest, but he passed the remaining portions of the examination. On the May 1989 examination, Petitioner received on his essay a score of two from each of the two readers who scored his essay. The four points he received on the essay portion of the subtest, when added to the score he received on the multiple choice portion of the subtest and statistically adjusted were insufficient to attain a passing grade on the communication subtest. Petitioner needed a score of three from each of the two readers to pass the communications subtest.

20. Petitioner retook the communications subtest in November 1989 and again failed to achieve a passing score. On the November 1989 examination, Petitioner again received on his essay a score of two from each of the two readers. Petitioner needed a score of three from one of the two readers to pass the communications subtest.

    
    

21. There was nothing about the testing conditions on either occasion that treated the Petitioner differently from any other candidate or that interfered with his opportunity to do his best writing. The holistic scoring process for each examination was carried out according to expectations without any problems or unusual circumstances. The procedural safeguards, designed to ensure uniformly fair scoring, were followed for both examinations. In neither case did the scoring of Petitioner's essay trigger the referring part of the process.

    
    

22. In the May 1989 examination, eight range finders were used, but only two dealt with the topic Petitioner selected. Those range finders were assigned a score of 2 and a 4, respectively.

    
    

23. Dr. Robert Blackmon and Dr. David Kirby, both of whom have impressive academic credentials and both of whom were accepted as experts in the field of holistic scoring of essays, were of the opinion that the Petitioner's essay on the May 1989 examination merited a score of three. Prior to forming that opinion, each had reviewed the two range finders pertinent to the topic selected by Petitioner and the scoring criteria used by the readers. Each then evaluated Petitioner's essay based on their review of the range finders and the criteria.

    
    

24. In the November 1989 examination, six range finders were used, with two dealing with the topic selected by Petitioner. Following the procedure they used in evaluating the May 1989 essay, Dr. Blackmon and Dr. Kirby reviewed Petitioner's essay for the November 1989 exam. Again, both of these witnesses were of the opinion that the Petitioner's essay merited a score of three.

    
    

25. The scoring of an essay using only two range finders is not as accurate as the scoring would be if there were additional range finders to be used for comparison. Neither Dr. Blackmon or Dr. Kirby reviewed range finders from other administrations of the FELE, neither reviewed the samples that had been selected, and neither had participated in any process designed to build consensus as to the appropriate scoring technique. Both reviewed Petitioner's two essays in greater detail than that employed by the holistic scoring contemplated for the FELE examination.

    
    

26. The use of only two range finders on the topic selected by Petitioner for the May 1989 exam and for the November 1989 exam did not invalidate the scoring process. The readers involved in the scoring of Petitioner's essays for the May and November 1989 examinations had sufficient range finders and sufficient samples to enable them to fairly score Petitioner's essays. While Petitioner's experts were of the opinion that a higher score should be awarded, these opinions were formed using a pre-scoring process different than that used by Respondent in evaluating the FELE and are not afforded greater weight than the scores given by Respondent's readers.

    
    

27. It is found, based on the greater weight of the evidence, that Petitioner's essays were fairly scored by Respondent's readers, that the scoring process used by Respondent complied in all material ways with the holistic scoring method, and that the scores awarded Petitioner by Respondent's readers should not be changed.

    CONCLUSIONS OF LAW

    
    

28. The Division of Administrative Hearings has jurisdiction over this matter. Section 120.57(1), Florida Statutes.

    
    

29. Petitioner has the burden of proving by a preponderance of the evidence that he is entitled to the relief he seeks. Rule 28-6.08(3), Florida Administrative Code. See also, Florida Department of Transportation v. J.W.C., Co., 396 So.2d 778 (Fla. 1st DCA 1981). Petitioner has failed to meet that burden.

RECOMMENDATION

Based upon the foregoing Findings of Fact and Conclusions of Law, it is RECOMMENDED that a Final Order be entered which denies Petitioner's

challenge to the scoring of his essay on the May 1989 Florida Educational Leadership Examination and which denies his challenge to the scoring of his essay on the November 1989 Florida Educational Leadership Examination.

DONE AND ORDERED in Tallahassee, Leon County, Florida, this 31st day of July, 1991.

CLAUDE B. ARRINGTON

Hearing Officer

Division of Administrative Hearings The DeSoto Building

1230 Apalachee Parkway

Tallahassee, Florida 32399-1550

(904) 488-9675

Filed with the Clerk of the Division of Administrative Hearings this 31st day of July, 1991.

ENDNOTES

1/ Dr. Charles R. Blackmon, one of Petitioner's expert witnesses, expressed the opinion that the readers were not qualified because of their lack of experience in school administration. Dr. Blackmon's opinion in this regard is rejected as being contrary to the greater weight of the evidence. Likewise rejected is Petitioner's challenge to the qualifications of the readers because no statistics were used to evaluate their performances. The qualifications of the readers were established by the testimony presented at the hearing and by the exhibits which were accepted into evidence.

2/ Petitioner does not challenge Respondent's use of the holistic method of evaluating his essays. His challenge is limited to the scores that resulted by the application of the scoring method.

3/ The readings for the May 1989 and November 1989 FELEs were not large enough to justify the use of table leaders. If table leaders are necessary, those individuals would be chosen and would typically participate in the selection of range finders and samples.

APPENDIX TO THE RECOMMENDED ORDER IN CASE NO. 90-2164

The following rulings are made on the proposed findings of fact submitted on behalf of the Petitioner.

1. The proposed findings of fact in paragraphs 1, 2, 3, 4, 5, 6, 11, 12, 13, 16, and 20 are adopted to the extent the proposed findings are deemed necessary to the conclusions reached.

2. The proposed findings of fact in paragraphs 7, 8, 10, 14, 15, 17, 19, and 22 are adopted in material part by the Recommended Order.

3. The proposed findings of fact in paragraph 9 are rejected because the proposed finding is not necessary to the conclusions reached and to protect the confidentiality of the examination question.

4. The proposed findings of fact in paragraphs 18 and 21 are rejected as being subordinate to the findings made.

5. The proposed findings of fact in the first sentence of paragraph 23 are rejected as being the recitation of testimony that is without probative value. The proposed findings of fact in the second sentence of paragraph 25 are rejected as being subordinate to the findings made.

The following rulings are made on the proposed findings of fact submitted on behalf of the Respondent.

1. The proposed findings of fact in paragraphs 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 13, 14, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 30, and 34 are adopted in material part by the Recommended Order.

2. The proposed findings of fact in paragraphs 7, 10, and 15 are adopted in part by the Recommended Order and are rejected in part as being subordinate to the findings made.

3. The proposed findings of fact in paragraphs 28, 29, 31, and 33 are rejected as being subordinate to the findings made.

COPIES FURNISHED:

Thomas W. Young, III, Esquire FEA/United

118 North Monroe Street Tallahassee, Florida 32399-1700

Charles S. Ruberg, Esquire Assistant General Counsel Florida Board of Education The Capitol - Suite 1701

Tallahassee, Florida 32399-0400

George A. Bowen

Acting Executive Director

301 Florida Education Center

325 W. Gaines Street Tallahassee, Florida 32399-0400

Jerry Moore, Administrator Professional Practices Service

352 Florida Education Center

325 W. Gaines Street Tallahassee, Florida 32399-0400

NOTICE OF RIGHT TO SUBMIT EXCEPTIONS

All parties have the right to submit written exceptions to this Recommended Order. All agencies allow each party at least 10 days in which to submit written exceptions. Some agencies allow a larger period within which to submit written exceptions. You should contact the agency that will issue the final order in this case concerning agency rules on the deadline for filing exceptions to this Recommended Order. Any exceptions to this Recommended Order should be filed with the agency that will issue the final order in this case.

Docket for Case No: 90-002164

Orders for Case No: 90-002164