STATE OF FLORIDA

DIVISION OF ADMINISTRATIVE HEARINGS

ANA C. RIVERO, )

)

Petitioner, )

)

vs. ) Case No. 02-1928

)

DEPARTMENT OF HEALTH, )

)

Respondent. )

)

RECOMMENDED ORDER

Pursuant to notice, a formal hearing was held in this case on September 20, 2002, by video teleconference, with the Petitioner appearing in Miami, Florida, and the Respondent appearing in Tallahassee, Florida, before Patricia Hart Malono, a duly-designated Administrative Law Judge of the Division of Administrative Hearings, who presided in Tallahassee, Florida.

APPEARANCES

For Petitioner: Dr. Ana Rivero, pro se

5787 Southwest 88 Court

Miami, Florida 33173

For Respondent: E. Renee Alsobrook, Esquire

Cassandra Pasley, Esquire Department of Health

4052 Bald Cypress Way Bin A02

Tallahassee, Florida 32399-1703

STATEMENT OF THE ISSUE

Whether the Petitioner should receive a passing grade on the Clinical Application of Medical Knowledge and Basic Science & Disease Process portions of the Florida Medical Licensure Examination ("FMLE") administered November 15 and 16, 2001.

PRELIMINARY STATEMENT

In Examination Grade Reports dated March 1, 2002, the Department of Health ("Department") notified Dr. Ana Rivero that she had failed to achieve passing scores on the Clinical Application of Medical Knowledge and Basic Science & Disease Process portions of the FMLE administered November 15 and 16, 2001. The Department reported that Dr. Rivero achieved a score of 331 points on the Clinical Application of Medical Knowledge portion of the examination, which had a minimum passing score of

350 points, and that Dr. Rivero achieved a score of 332 points on the Basic Science & Disease Process portion of the examination, which had a minimum passing score of 350 points. Dr. Rivero timely requested an administrative hearing, and the Department transmitted the matter to the Division of Administrative Hearings for assignment of an administrative law judge.

In her request for an administrative hearing, Dr. Rivero challenged the fairness of the administration of the November 15 and 16, 2001, FMLE on four grounds, as follows:

1. On the first day of the FMLE, the answer sheets were not distributed on time. They were distributed 45 minutes after the exam began; consequently not only causing a lot of stress on myself and other test takers, but also prolonging the termination of the exam.

   
   

2. At random, I observed as I took the exam, some questions whose correct answer was not the same on the English version as on the Spanish version, yet there is only one answer sheet for both English and Spanish exams.

   
   

3. Under Section 458.3115 1(d). . . States "Every person eligible for restricted licensure under this section may sit for the USMLE [United States Medical Licensure Examination] or the agency and board- developed examination five times within five calendar years." I have only taken the FMLE four times in a three-year period beginning in 1999.

   
   

4. I also question the Angoff method, which is the chosen method of scoring the FMLE, because it differs from the method used to score the USMLE.

At the hearing, Dr. Rivero testified in her own behalf, and Petitioner's Composite Exhibit 1 was offered and received into evidence without objection; this exhibit consists of six separate documents, including the Examination Grade Reports for the Clinical Application of Medical Knowledge and Basic Science & Disease Process portions of the FMLE administered November 15 and 16, 2001; a Department of Health Election of Rights Form, together with a notice that the final FMLE examination was administered November 15, 2001, and that it cannot be given

again; a copy of Section 458.3115, Florida Statutes (2002); a copy of Dr. Rivero's request for an administrative hearing, to which is attached a memorandum from Department employee Gregory Stone; a letter to Dr. Rivero from David Paulson, Manager of the Department's Testing Services, in which he explains the scoring system for the FMLE; and a copy of "Scoring and Score Reporting" dated September 16, 2002, from the 2002 USMLE Bulletin. The Department presented the testimony of Eunice Filer, a psychometrician employed by the Department; Margaret Anglin, a Program Administrator with the Department; and Jeanne Smith, an Examination Supervisor with the Department. Respondent's Exhibits 1 through 3 were offered and received into evidence.1

One of the bases for Dr. Rivero's challenge to the examination is that there were discrepancies between the English and the Spanish versions in the order in which the answers were listed for some questions in the examination. Dr. Rivero wanted access to the Spanish version of the examination so that she could compare the Spanish version with the English version to identify the discrepancies. The Department objected to production of the entire examination and requested that

Dr. Rivero's review be limited to those questions and answers that the Department scored as incorrect on Dr. Rivero's examination. The undersigned limited the production as requested by the Department, and the Department agreed to

provide the English and Spanish versions of the selected questions and answers to Dr. Rivero for her review. The parties were advised that Dr. Rivero could request that the final hearing be reconvened if she wished to present evidence to establish a discrepancy between the English and Spanish versions of the examination.

A telephone conference was held on November 4, 2002, with counsel for the Department and Dr. Rivero in attendance. At that time, Dr. Rivero stated that she had not been given access to the English and Spanish versions of all of the examination questions and answers and that she had refused to review the examination questions and answers produced for her review by the Department in accordance with the instructions of the undersigned. Dr. Rivero stated during the telephone conference that she did not wish to present any additional evidence in support of her challenge to the examination, and the hearing was closed.

The transcript of the proceedings conducted on September 20, 2002, was filed with the Division of Administrative Hearings on October 18, 2002; during the

November 4, 2002, telephone conference, it was agreed that the parties would file their proposed findings of fact and conclusions of law on or before November 22, 2002. Both parties

timely filed their proposals, which have been considered in the preparation of this Recommended Order.

FINDINGS OF FACT

Based on the oral and documentary evidence presented at the final hearing and on the entire record of this proceeding, the following findings of fact are made:

1. The Department is the state agency responsible for licensing and regulating physicians practicing medicine in Florida, including foreign-licensed physicians.

   Sections 458.311 and 458.3115, Florida Statutes (2001);

   
   

   Rule 64B8-5.002, Florida Administrative Code. The Department is also authorized to administer licensing examinations to physicians seeking to practice medicine in Florida.

   Section 456.017, Florida Statutes (2002).

   
   

2. Dr. Rivero was accepted as a candidate for the FMLE and sat for the examination on November 15 and 16, 2001. Dr. Rivero attained a scaled score of 332 points on the Basic Sciences & Disease portion of the examination and 331 points on the Clinical Application of Medical Knowledge portion of the examination. Each of these portions of the FMLE administered November 15 and 16, 2001, had a minimum passing score (also known as "cut score") of 350 points.

3. On November 15, 2001, the first day of the examination, the Department staff who were to administer the examination were

   notified that the Federal Express shipment of examination materials was incomplete and did not include the laboratory value sheets and the answer sheets to be used for each portion of the examination. The supervisor of the examination administration arranged to have copies of the appropriate documents available that morning.

4. The examination candidates, who had been told to arrive at the examination site at 7:30 a.m., were not admitted to the examination room until 8:30 a.m. as a result of the problem with the laboratory value and answer sheets. The candidates were told to skip the questions that required use of the laboratory value sheets and to write the answers in the examination booklets for the questions that required use of the answer sheets. The candidates were advised prior to beginning the examination that they would be allowed additional time to transfer their answers from the booklet to the answer sheet.

5. The examination began at 9:30 a.m. on November 15, 2001, after a delay of one hour. The administration supervisor made an error calculating the time and gave the candidates four hours and ten minutes to complete the examination, rather than the prescribed four hours. In addition, all candidates who wanted additional time to transfer their answers from the examination booklet to their answer sheets were given as much additional time as necessary.

6. Dr. Rivero experienced stress and nervousness as a result of the delay and confusion in the administration of the examination that might have affected her performance on the examination. She did, however, have sufficient time to complete the examination on November 15, 2001, and to transfer her answers to the answer sheet.

7. The minimum passing score on both portions of the examination was 350 points. These "cut scores" were developed for the November 2001 FMLE using the Angoff method of scoring. The Angoff Method is a widely used method for selecting the "cut score" for an examination. For each administration of the FMLE, a group of physicians are chosen to review the examination and determine, question by question, the percentage of minimally competent people who would answer each question correctly. The "cut score" for each portion of the examination is developed by averaging the responses of the physicians.

8. The Angoff method was a valid methodology for ascertaining the "cut scores" for the November 2001 administration of the FMLE.

9. After the examination was scored, a group of physicians and a psychometrician met to review all of the questions that were the subject of a complaint by examination candidates and all of the questions that a statistically significant number of candidates answered incorrectly. The group also conducted a

   Point by Serial review of the examination, which involves establishing that the candidates scoring highest on the examination answered a particular question correctly, while candidates scoring lowest on the examination answered the same question incorrectly.

10. As part of this post-examination review, the November 2001 FMLE was reviewed for discrepancies between the

    order of the answers to questions in the English version of the examination and the order of the answers to questions in the Spanish version of the examination. No discrepancies were found.2 In addition, Dr. Rivero conceded that there were no discrepancies between the English and Spanish versions of the questions she answered incorrectly.

11. The results of the review of the November 2001 FMLE established that the examination was fair, reliable, and valid.

12. The November 2001 FMLE was developed, scored, and reviewed in accordance with the procedures normally used by the Department. Dr. Rivero has failed to establish that she should be awarded additional credit for any question the Department scored as incorrect on the Basic Sciences & Disease and on the Clinical Application of Medical Knowledge portion of the examination.3

    CONCLUSIONS OF LAW

    
    

13. The Division of Administrative Hearings has jurisdiction over the subject matter of this proceeding and of the parties thereto pursuant to Sections 120.569 and 120.57(1), Florida Statutes (2002).

14. Dr. Rivero has the burden of proving by a preponderance of the evidence that the Department's administration and scoring of the November 15 and 16, 2001, FMLE was arbitrary or capricious or constituted an abuse of discretion. See State ex rel. Glasser v. J. M. Pepper, 155 So. 2d 383 (Fla. 1st DCA 1963); State ex rel. Topp v. Board of Electrical Examiners, 101 So. 2d 583 (Fla. 1st DCA 1958). Based on the facts found herein, Dr. Rivero has failed to meet this burden.

15. The record reflects that the Department made accommodations to the examination candidates on November 15, 2001, to allow them to compensate for the late arrival of the laboratory value sheets and answer sheets. Dr. Rivero did not establish that she suffered any adverse effects of the delay except for an increase in her level of stress and nervousness. This is insufficient to establish that she should be awarded a passing score on the Clinical Application of Medical Knowledge and Basic Science & Disease Process portions of the FMLE administered November 15, and 16, 2001.

16. Dr. Rivero conceded that, even if there had been discrepancies between the English and Spanish versions of the examination in the listing of answers to several questions, there were no such discrepancies with respect to the questions that the Department scored as incorrect on her examination. Therefore, any such discrepancies would not result in

    Dr. Rivero's being awarded a passing score on the Clinical Application of Medical Knowledge and Basic Science & Disease Process portions of the FMLE administered November 15, and 16, 2001.

17. Finally, the evidence presented by Dr. Rivero is not sufficient to establish that the Department's use of the Angoff Method for developing the "cut scores" for the November 15 and 16, 2001, administration of the FMLE was arbitrary or capricious. Rather, the evidence establishes that the Angoff Method is widely used and produces an accurate "cut score." Consequently, Dr. Rivero has failed to establish that the "cut scores" for the examination should be lowered to a level that would result in her achieving a passing score of the Clinical Application of Medical Knowledge and Basic Science & Disease Process portions of the FMLE administered November 15, and 16, 2001.

RECOMMENDATION

Based on the foregoing Findings of Fact and Conclusions of Law, it is RECOMMENDED that the Department of Health enter a final order dismissing the petition of Ana Rivero challenging her failing scores on the Clinical Application of Medical Knowledge and Basic Science & Disease Process portions of the FMLE administered November 15, and 16, 2001.

DONE AND ENTERED this 20th day of December, 2002, in Tallahassee, Leon County, Florida.

PATRICIA HART MALONO

Administrative Law Judge

Division of Administrative Hearings The DeSoto Building

1230 Apalachee Parkway

Tallahassee, Florida 32399-3060

(850) 488-9675 SUNCOM 278-9675

Fax Filing (850) 921-6847 www.doah.state.fl.us

Filed with the Clerk of the Division of Administrative Hearings this 20th day of December, 2002.

ENDNOTES

1/ Respondent's Exhibits 2 and 3 are the Examination Grade Reports at issue herein, and these two documents are also included in Petitioner's Composite Exhibit 1 and were transmitted by the Department to the Division of Administrative Hearings with Dr. Rivero's request for an administrative hearing. The Department claims that these documents are confidential pursuant to Section 456.014(2), Florida Statutes; this contention is rejected, however, because that statutory section applies only to "examination questions and answers." The documents identified by the Department do not contain any examination questions or answers.

2/ One of the exhibits introduced into evidence by Dr. Rivero is a memorandum dated April 1, 2002, from Gregory Stone, a Senior Management Analyst, to Cherry Shaw, the Department's former attorney in this case, in which he addresses Dr. Rivero's complaints regarding the November 2001 examination. In the memorandum, Mr. Stone stated that "two items were translated incorrectly on the Spanish version of the examination" and that "all candidates were given credit for the questions." Eunice Filer, the Department's psychometrician, testified that this information is incorrect, that Mr. Stone had confused the November 2001 FMLE with the FMLE administered on another date.

3/ A final point raised by Dr. Rivero in her request for an administrative hearing is that she has taken the FMLE only four times within the past five years. Dr. Rivero is apparently referring to the notification she received advising her that, pursuant to Section 456.017(1)(c)2., Florida Statutes (2002), the final administration of the FMLE was November 15, 2001, and that the examination cannot be given again.

Dr. Rivero cites Section 548.3115(1)(d), Florida Statutes (2002), as providing her the right to sit for the USMLE or the FMLE five times within five years. The question of whether

Dr. Rivero can sit for the FMLE or the USMLE again is not within the purview of this administrative hearing. Nonetheless, it is noted that the evidence reflects that Dr. Rivero sat for the FMLE in May 1999, May 2000, May 2001, and November 2001; she applied for the November 1999 administration of the FMLE, but she did not show up to take the test. Dr. Rivero also sat for the USMLE in March 1998 and June 1998. Dr. Rivero has been unsuccessful in her efforts to pass either the FMLE or the USMLE.

COPIES FURNISHED:

E. Renee Alsobrook, Esquire Department of Health

4052 Bald Cypress Way Bin A02

Tallahassee, Florida 32399-1703

Ana C. Rivero

5787 Southwest 88 Court

Miami, Florida 33173

William W. Large, General Counsel Department of Health

4052 Bald Cypress Way Bin A02

Tallahassee, Florida 32399-1701

R.S. Power, Agency Clerk Department of Health 4052 Bald Cypress Way Bin A02

Tallahassee, Florida 32399-1701

NOTICE OF RIGHT TO SUBMIT EXCEPTIONS

All parties have the right to submit written exceptions within

15 days from the date of this recommended order. Any exceptions to this recommended order should be filed with the agency that will issue the final order in this case.

Docket for Case No: 02-001928

Orders for Case No: 02-001928