JOHN T. CURTIN, District Judge.

The City of Buffalo has filed a motion (Item 580) for final relief under the operative interim remedial hiring order in this case, followed by termination of that order, based on the development of a written entry-level firefighter examination that meets the applicable legal standards for validation of selection procedures to be utilized for appointments to the Buffalo Fire Department Training Academy (the "Fire Academy"). The United States does not oppose the City's request for this relief. Members of Color Helping All Society, Inc. ("MOCHA"), plaintiff-intervenor, has filed a memorandum (Item 589) in opposition to the motion.

For the reasons that follow, the City's motion is granted.

BACKGROUND

In August 1978, this court found after trial that the City's Police and Fire Departments had engaged in a pattern or practice of employment discrimination against blacks, Spanish-surnamed Americans ("SSA's"), and women, in violation of Title VII of the Civil Rights Act of 1964 (as amended by the Equal Employment Opportunity Act of 1972), 42 U.S.C. § 2000e et seq. ("Title VII"). See U.S. v. City of Buffalo, 457 F.Supp. 612 (W.D.N.Y. 1978). Upon considering the parties' proposals for a remedy to adequately address the effects of past discrimination, the court entered a final remedial consent decree on November 23, 1979 (the "Remedial Decree") which, among other forms of injunctive relief, ordered the City to adopt and seek to achieve an interim goal of making 50% of all entry-level police and firefighter appointments from among qualified Black and SSA applicants.1 These "one-for-one" interim hiring requirements were to remain in effect until "the City has satisfied the court that it has developed valid selection procedures" in accordance with the Equal Employment Opportunity Commission ("EEOC") Uniform Guidelines on Employee Selection Procedures ("Uniform Guidelines"), 29 C.F.R. Part 1607. U.S. v. City of Buffalo, 633 F.2d 643, 648-49 (2d Cir. 1980) (modifying November 23, 1979 Remedial Decree by striking provision directing City to seek to achieve "long term goal" of reaching minority composition of police officers and firefighters comparable to that of City's work force as a whole according to census; affirming as modified).

For the next ten years, the recruitment and hiring needs of the City's Police and Fire Departments were met in substantial compliance with these interim requirements. Then, in September 1989, upon the City's presentation of evidence demonstrating that the minority composition of each department sufficiently reflected the minority composition of Buffalo's civilian labor force, the court supplanted the 50% interim hiring requirements with the following provisional hiring order, which has come to be known as the "Applicant Flow Order:"

U.S. v. City of Buffalo, 721 F.Supp. 463, 469 (W.D.N.Y. 1989), aff'd, 993 F.2d 1533 (2d Cir. 1993). Subsequent to the entry of this order, the City has made several attempts to develop valid selection procedures for entry-level firefighters in accordance with applicable legal standards, without success,3 and a number of Fire Academy classes have been convened under the requirements of the Applicant Flow Order.

In January 2013, after soliciting proposals from several nationally recognized test developers, the City retained EB Jacobs, Inc., of State College, PA, to develop and administer a written examination for the entry-level position of firefighter. According to the Declaration of CEO Rick Jacobs, Ph.D., the EB Jacobs Fire Service Aptitude Battery ("FSAB"), administered by the City on November 19, 2013 (the "2013 Exam"), was developed utilizing a comprehensive job analysis showing a strong relationship between test content and the requirements of the job of firefighter in the City of Buffalo (the "2013 Job Analysis"), and criterion-related studies showing positive correlation with job performance, in accordance with the requirements of Title VII and the Uniform Guidelines. See generally Item 580-5 (Declaration of Rick Jacobs).4

A total of 2,427 candidates took the 2013 Exam, representing a 78.6% increase in the overall candidate pool as compared to the last previous administration of the firefighter exam in 2008 (the "2008 Exam"). Id. at ¶ 11. The candidate pool was also more diverse: 39.9% of the candidates who took the 2013 Exam were Black, an increase of 80.5% over the 2008 Exam; 8.0% were Hispanic, an increase of 3.9%; and 14.7% were women, an increase of 75%. Id. According to the City, these increases are attributable in large part to an expanded recruitment program which placed special emphasis on social media and outreach to local community groups and churches. See Item 580-1 (Declaration of Carolyn A. Lenczyk, Acting Administrative Director of Civil Service for the City of Buffalo), ¶ 4; see also Item 590-2 (Declaration of Gladys "G.G." Herndon-Hill, Commissioner for Human Resources for the City of Buffalo), ¶ 6.

Using a scoring model developed in collaboration with the City and the U.S. Department of Justice ("DOJ"), EB Jacobs provided the City with a list of candidates ranked in descending order based on their performance on the 2013 Exam. On June 4, 2014, the City adopted the 2014 Civil Service Eligibility List for the entry-level firefighter position, listing 1,929 candidates who passed the 2013 Exam. Item 580-4. As indicated in the court's order entered February 23, 2015 (Item 581) granting the City's request for an interim stay of the Applicant Flow Order pending determination of this motion, the City anticipates hiring in excess of 250 entry-level firefighters over the four-year life of the 2014 Eligibility List, due to the large number of current vacancies, expected retirements, and other attrition.

The City now seeks an order terminating the requirements of the Applicant Flow Order, and allowing future appointments to the Fire Academy to be made by rank order under the requirements of New York Civil Service Law, on the ground that the City has finally developed a written examination for the entry-level firefighter position that meets the legal standards for validating employment selection procedures having adverse impact on minority applicants. See Item 580-7. Relying primarily on the Jacobs Declaration,5 the City contends that there is ample evidence for the court to find that the proposed use of the 2013 Exam is a valid selection procedure that is predictive or significantly correlated with important elements or work behavior relevant to the job of firefighter in the Buffalo Fire Department. The City also seeks a final order of relief under the Applicant Flow Order requiring the appointment of four Black candidates and one Hispanic candidate from the 2014 Eligibility List, in addition to those who would otherwise be appointed in rank order, as "makeup appointments" to account for the minority applicants who did not complete the last Fire Academy training class in April 2012. As indicated, the United States has joined the City's requests for final relief under, and termination of, the Applicant Flow Order.

For its part, MOCHA contends that the validity evidence relied upon by the City is too tenuous to support rank-order hiring, and proposes banded scoring as a means of providing more equitable seniority to Black candidates. Alternatively, MOCHA contends that if the court grants the City's motion, the relief ordered should be conditioned upon various measures to monitor any discriminatory treatment of minority candidates in subsequent steps of the selection procedure for entry-level firefighters. See Item 589.

For the reasons that follow, the City's motion is granted.

DISCUSSION

I. Motion to Vacate the Applicant Flow Order: Validity of the 2013 Exam

In order to grant the relief requested by the City, the court must determine whether the evidence presented by Dr. Jacobs, the City's test developer and validation expert, is sufficient to demonstrate that the City's proposed use of the 2013 Exam results for rankorder hiring is a valid employment selection procedure. As Dr. Jacobs advises, since the final statistics on the adverse impact of rank-order hiring from the 2014 Eligibility List will not be available until the list expires, compliance with the Uniform Guidelines' "four-fifths" adverse impact hiring standard6 must be estimated based on the Fire Department's anticipated hiring of more than 250 entry-level firefighters over the four-year life of the List. Given the Fire Department's past experience of processing, on average, four candidates to yield one successful recruit, the City will reach well beyond the top 40% of candidates who completed the 2013 Exam (corresponding with the 971st candidate on the List), with the adverse impact ratio reaching and remaining at four-fifths (80%) for women, Hispanics, and Blacks at 0.2%, 27.1%, and 37.8%, respectively, of all candidates. See Item 580-5, ¶¶ 25-26. These estimates, accepted by the court and all parties for the purposes of deciding the pending motions, support the conclusion that the City's proposed use of the 2013 Exam will not result in a violation of the Uniform Guidelines' four-fifths rule.

The federal courts have required an employer seeking validation of an employment qualification examination to show, by professionally acceptable methods, that the examination is "predictive of or significantly correlated with important elements of work behavior which comprise or are relevant to the job or jobs for which candidates are being evaluated." Albemarle Paper Co. v. Moody, 422 U.S. 405, 431 (1975) (internal quotation marks omitted). "The touchstone is business necessity. If an employment practice which operates to exclude [minorities] cannot be shown to be related to job performance, the practice is prohibited." Griggs v. Duke Power Co., 401 U.S. 424, 431 (1971).

In the Second Circuit, the holding in Guardians Ass'n v. Civil Serv. Comm'n of New York, 630 F.2d 79 (2d Cir. 1980), aff'd, 463 U.S. 1140, cert. denied, 463 U.S. 1228 (1983), has long provided the benchmark for courts charged with assessing the validity of an employment examination. According to Guardians, the threshold task in determining validity is to select the appropriate method for assessing job-relatedness from the three basic methods recommended by the Uniform Guidelines: criterion-related validation, content validation, and construct validation. Id. at 91-92 (citing 29 C.F.R. §§ 1607. 5(B), 1607.14). As described generally in the Guidelines:

29 C.F.R. § 1607. 5(B).

Guardians recommends a "cautionary approach" to validation under the Guidelines, Guardians, 630 F.2d at 93, reminding courts that "the Guidelines are not administrative regulations promulgated pursuant to formal procedures established by Congress. They are entitled to deference, not obedience." Id. at 91 (internal quotation marks omitted); see also Gulino v. New York State Educ. Dept., 460 F.3d 361, 383-84 (2d Cir. 2006) ("[A]lthough we approach the Guidelines with the appropriate mixture of `deference and wariness,' thirty-five years of using these Guidelines makes them the primary yardstick by which we measure" defendants' attempt to validate employment selection procedures) (quoting Guardians, 630 F.2d at 91).

In this case, Dr. Jacobs' Declaration reflects that EB Jacobs developed the 2013 Exam by first conducting the comprehensive 2013 Job Analysis in order to provide test developers a detailed understanding of the job of firefighter in the Buffalo Fire Department. The 2013 Job Analysis included interviews and observations of entry level firefighters assigned to the various department facilities throughout the City; formation of panels of incumbent Buffalo firefighters to provide updated information regarding the specific duties, tasks, and responsibilities of firefighters; and surveys of incumbents to collect detailed information regarding job tasks, job duties, job functions, equipment, cognitive abilities, physical abilities, and personal characteristics. See Item 580-5, ¶¶ 7-8; see also Item 580-6 (Summary of Results of the 2013 Job Analysis).

As explained by Dr. Jacobs, and as outlined in the discussion below, the strategy for establishing the validity of the FSAB, as administered in 2013, involved multiple approaches based on the recommendations of the Uniform Guidelines, including content validation; concurrent criterion-related validation; and predictive criterion-related validation. See Item 580-5, ¶ 16.

A. Content Validity

Under Guardians and the Uniform Guidelines, an employment examination has content validity "if the content of the examination matches the content of the job." Guardians, 633 F.2d at 242.

Id. (quoting Vulcan Soc. of New York City Fire Dept., Inc. v. Civil Service Commission of City of New York, 360 F.Supp. 1265, 1274 (S.D.N.Y.), aff'd, 490 F.2d 387 (2d Cir. 1973). As further explained in Guardians, the logical first step in developing a content valid test is a proper job analysis, defined as "a thorough survey of the relative importance of the various skills involved in the job in question and the degree of competency required in regard to each skill. It is conducted by interviewing workers, supervisors and administrators; consulting training manuals; and closely observing the actual performance of the job." Id. (quoting Vulcan, 360 F. Supp. at 1274); see also Uniform Guidelines, 29 C.F.R. § 1607.14C(2).

In this case, evidence of content validity was developed through the comprehensive 2013 Job Analysis, which was utilized by EB Jacobs to establish a relationship between the content of the FSAB and the requirements of the job of firefighter in the City of Buffalo. The FSAB contains three sections: a Cognitive Ability Test, measuring seven areas of cognitive ability important for successful performance of the firefighter job; a Work Styles Questionnaire, measuring twenty-six different personality attributes; and a Life Experience Survey, measuring seven attributes related to past work behaviors. Item 580-5, ¶ 12.

As reported in the Summary of Results for the 2013 Job Analysis, EB Jacobs staff examined a total of 83 Task, Equipment, and Ability/Characteristic Surveys submitted by incumbents, focusing on the incumbents' profiles of the relative importance of job duties, cognitive abilities, physical abilities, and personal characteristics in the successful performance of the job of firefighter in the City of Buffalo. These profiles were then compared to the "relative importance profiles" from EB Jacobs' 2008 Job Analysis Survey, showing similar results and confirming that the job and attributes necessary for success had not changed significantly since the last prior administration of the FSAB. The 2013 cognitive ability profile was also compared to cognitive ability profiles from job analyses conducted by EB Jacobs in recent years for fire service agencies in eight other jurisdictions, demonstrating a strong correlation between the cognitive abilities reported by incumbents in those agencies and incumbents in Buffalo as critical for success in the performance of core firefighting activity. Based on the findings from the 2013 Job Analysis, EB Jacobs recommended the use of the FSAB as an appropriate entry-level examination for selection of firefighters in the City of Buffalo. See Item 580-6, pp. 2-5.

B. Criterion-Related Validity

"Criterion-related validity studies correlate test scores with job performance." Guardians, 633 F.2d at 244. A criterion-related validity study is essentially "a comparison of how examinees perform on an exam with how they perform in the job itself." U.S. v. City of New York, 2012 WL 4503253, at *3 (E.D.N.Y. Sept. 28, 2012); see 29 C.F.R. § 1607.14(B). The study "should consist of empirical data demonstrating that the selection procedure is predictive of or significantly correlated with important elements of job performance." 29 C.F.R. § 1607.5(B).

Williams v. Ford Motor Co., 187 F.3d 533, 540 (6th Cir. 1999) (citing cases from the 5th, 9th, and 11th Circuits).

EB Jacobs' criterion-related validity study for the 2013 Exam utilized both "predictive validation" and "concurrent validation." These two methods are described generally in the Vulcan Society case as follows:

Vulcan Society, 360 F. Supp. at 1273.

As reflected in Dr. Jacobs' Declaration, the concurrent criterion-related validity study of the 2013 Exam involved an examination of the scoring models from recent administrations of the FSAB in a number of fire departments nationwide, including the 2008 administration of the FSAB in Buffalo, along with test and performance data collected concurrently from 260 incumbent firefighters in those fire departments, and the new candidate data from the 2013 Exam. See Item 580-5, ¶ 18. The predictive criterion-related study involved an examination of the data pertaining to approximately 150 firefighters hired in the four Buffalo Fire Academy classes convened based on the results of the 2008 Exam, including written examination results, practical skills testing results obtained from Fire Academy personnel, and job performance ratings developed by supervisory personnel working in conjunction with EB Jacobs staff. EB Jacobs then conducted a separate analysis to examine the relationship between this compiled criterion data and predictor data obtained from the FSAB cognitive ability test, work styles questionnaire, and life experience composite survey scales. See id. at ¶ 20. The positive validity coefficients associated with these studies, as reported in the Summary Table attached to the Jacobs Declaration (id. at p. 14), were reviewed by EB Jacobs staff in collaboration with representatives of the City and the Department of Justice ("DOJ") to explore alternative scoring solutions, ultimately resulting in the adoption of the scoring model alternative having the most favorable overall adverse impact ratio. Id. at ¶ 23. Based on the information reported in EB Jacobs' criterion-related validity study, the DOJ's technical expert, David Jones, Ph.D., corroborated the use of this scoring model as appropriate for rank-order selection of 2013 Buffalo Fire Academy candidates. See Item 590-3 (Reply Declaration of Dr. Jacobs), ¶ 3.

MOCHA contends that the validity coefficients reported by EB Jacobs are "questionable" because they appear to account for only a small percentage of the variation in the scores on the 2013 Exam, and therefore urges the court not to vacate the Applicant Flow Order without a further compelling showing by the City that the scores are in fact appropriate for rank-order use. However, as explained by Dr. Jacobs, the methods employed by EB Jacobs to generate the validity studies supporting the City's proposed use of the 2013 Exam, corroborated by the DOJ's expert, are consistent with accepted practices for evaluating selection procedures within the field of Industrial Psychology (see id. at ¶¶ 4-10), and MOCHA has submitted no expert testimony, professionally creditable studies, or other evidence to challenge these methods in a manner sufficient to require further expenditure of scarce public resources on an evidentiary hearing.

Moreover, as Dr. Jacobs explains at further length in his Reply Declaration, the validity coefficients cited by MOCHA understate the relationship between the test and the applicable performance criteria, since only firefighters hired from the four 2008 Academy classes—i.e., those individuals achieving higher scores on the 2008 Exam—could be included in the predictive criterion-related study that generated the validity coefficients. Once corrected to account for this "range restriction," the coefficient values are reported as significantly higher, and when considered along with the exam's "overall reliability coefficient" of 83%, the corrected validities provide further evidentiary support for finding a sufficient correlation between test scores and job performance. See id. at ¶¶ 3-5.

MOCHA also contends that scoring the Exam on a banded basis might provide more equitable seniority for Black candidates than strict rank-order use. However, as explained by Dr. Jacobs, the validation studies included a collaborative evaluation of psychometrically-based banding for consideration as a scoring model, and the DOJ's experts concurred with EB Jacobs staff that the strength of the validity coefficients from the local criterion-related validity study, combined with the strength of the test's overall reliability coefficient, outweighed the value of band scoring. See id. at ¶¶ 11-14. Again, MOCHA has not presented any expert evidence to refute these collaborative findings and conclusions, or to otherwise justify rejecting the parties' chosen scoring model.

Finally, MOCHA proposes that if the court grants the relief sought by the City and vacates the Applicant Flow Order, the relief should be conditioned upon modifying the final Remedial Order to provide for continued court supervision in order to assure that the City: maintains the same protocol for scoring accepted as valid for rank-order selection of candidates; continues the same recruitment protocol utilized to achieve the excellent minority turnout for the 2013 Exam; and conducts post-examination screening, training, and evaluation processes and procedures in a manner that minimizes adverse impact. See Item 589. While these proposals undoubtedly merit consideration, the court notes that vacating the Applicant Flow Order at this time does not terminate the action, or effect the court's exercise of jurisdiction as necessary or appropriate to insure and enforce rights to equal employment opportunities within the Buffalo Fire Department, including continuing supervision of the City's selection, screening, training, and other procedures affecting the employment of entry-level firefighters.

To be sure, and to reiterate:

U.S. v. City of Buffalo, 721 F. Supp. at 468-69. Considering the substantial progress made toward this final goal through the development, administration, scoring, and utilization of the 2013 Exam as a valid selection procedure, the court is confident that further collaborative efforts by the parties can at long last accomplish full compliance with the Remedial Decree in a manner that will ensure the City's continued non-discriminatory appointment of firefighter candidates without the need for federal court oversight.

Accordingly, upon review of the ample evidence of professionally acceptable content and criterion-related validation, and in the absence of any countervailing expert evidence of non-validity, the court accepts the joint representation of the City and the United States that the content of the 2013 Exam "is representative of important aspects of performance on the job for which the candidates are to be evaluated," and that the Exam is "predictive of or significantly correlated with important elements of work behavior which comprise or are relevant to the job" of entry-level firefighter in the City of Buffalo Fire Department. The court therefore terminates the requirements of the Applicant Flow Order, allowing future appointments to the Fire Academy to be made by rank order under the requirements of New York Civil Service Law.

II. Motion for Final Relief Under the Applicant Flow Order

The City also seeks a final order of relief under the Applicant Flow Order, which requires that applicants who did not survive the Fire Academy are not to be counted toward fulfillment of the applicant flow requirement. As outlined in the Lenczyk Declaration, the City appointed candidates from the 2008 Eligible List to four Fire Academy classes in accordance with the Applicant Flow Order, but four Black candidates and one Hispanic candidate did not complete the April 2012 Fire Academy. Item 580-1, ¶ 3; Item 580-3.7 The City therefore requests that, prior to vacating the Applicant Flow Order, the court enter an order requiring the "makeup appointments" of four Black candidates and one Hispanic candidate, in addition to those who would otherwise be appointed in rank order, from the 2014 Eligibility List. As with the motion to terminate the requirements of the Applicant Flow Order, the United States has joined the City's request for this final relief.

Accordingly, in the absence of any stated objection to the relief requested by this motion, and for the reasons set forth in the Lenczyk Declaration and accompanying submissions, the motion is granted.

CONCLUSION

For the foregoing reasons, the court finds that the validation evidence presented by the City, and corroborated by the United States, is sufficient to demonstrate that the entry-level firefighter examination administered by the City on November 19, 2013 is "predictive of or significantly correlated with important elements of work behavior which comprise or are relevant to the job" of firefighter in the City of Buffalo Fire Department. The court therefore grants the City's motion (Item 580) to terminate the interim hiring requirements of the Applicant Flow Order, allowing future appointments to the Fire Academy to be made by rank order in accordance with New York Civil Service Law. The court also grants the City's request for final relief under the Applicant Flow Order, requiring the appointment of four Black candidates and one Hispanic candidate from the 2014 Eligible List, in addition to those candidates who would otherwise be appointed in rank order.

The parties are directed to submit to the court, within 30 days of the date of entry of this decision and order (or within a mutually agreed-upon reasonable time frame), a status report addressing the remaining steps to be taken toward full compliance with the Remedial Decree and termination of the court's jurisdiction over this matter.

So ordered.