ORDER GRANTING IN PART AND DENYING IN PART DEFENDANTS' MOTIONS TO EXCLUDE THE PROFFERED EXPERT REPORTS OF CHARLES D. COWAN AND RICHARD K. GREEN

MARIANA R. PFAELZER, District Judge.

I. INTRODUCTION

Defendants move to exclude as inadmissible seven expert reports submitted by Plaintiffs regarding a proposed sampling methodology for loans backing residential mortgage-backed securities involved in the above-captioned cases. Pursuant to a stipulation of the parties, the Court ordered consolidated briefing in the six above-captioned cases and a hearing on the motion to exclude. The motion was fully briefed by the parties and submitted on September 27, 2013. The Court heard oral arguments on October 15, 2013. Having reviewed the motion, the response, the reply, and all related papers, including the expert reports, the rebuttal expert report, and the supplemental expert reports, the Court GRANTS IN PART and DENIES IN PART the motion to exclude.

II. BACKGROUND

These securities actions concern residential mortgage-backed securities ("RMBS") purchased by Plaintiffs in multiple offerings structured and sold by Countrywide entities, including Countrywide Financial Corporation, Countrywide Securities Corporation; Countrywide Capital Markets, LLC; Countrywide Home Loans, Inc.; CWALT, Inc.; CWABS, Inc.; CWMBS, Inc.; and CWEHQ, Inc. (collectively "Countrywide") between 2005 and 2007. The complaints allege federal and state causes of action against the Defendants.

RMBS are created through securitization. Securitization involves the creation of pools of residential mortgage loans, each of which produce cash-flows from the payment on the loans. The rights to the cash-flows of these pools are sold to investors as certificates. First, an originator issues residential mortgage loans, which are then pooled together by the originator or an acquirer. The loans are sold to depositors, who then transfer the loans to trusts pursuant to a pooling and servicing agreement. The trusts issue separate securities in the form of certificates for purchase by investors. The certificate entitles the holder to a portion of the cash-flow from the pool of underlying mortgages, called the supporting loan group ("SLG"). The certificates are sold in tranches, slices of the loan pool with different priorities of payment, interest rates and credit protection. As part of the securitization process, the tranches are assigned credit ratings by the credit rating agencies. For the RMBS in these cases, Countrywide served as an originator for some loans and acquired other loans from other originators. In addition, Countrywide served as the issuer for these RMBS.

In order to prove various allegations in the complaints, Plaintiffs have proposed the use of statistical sampling of a representative selection of loans from the SLGs supporting the RMBS purchased by the Plaintiffs as well as a sampling of a selection of all loans in the SLGs for all RMBS offered by Countrywide ("Countrywide RMBS") between 2005 and 2007. In accordance with the Court's Order Regarding Discovery Schedule (Doc. No. 125), the Defendants have moved to exclude the expert testimony regarding the initial methodology for the loan sampling proposed by Plaintiffs for failure to comply with Federal Rule of Evidence 702.

III. LEGAL STANDARD

Federal Rule of Evidence 702 governs the admissibility of expert testimony. Rule 702 states:

FED.R.EVID. 702. In applying Rule 702, the Court functions as a gatekeeper, determining whether proffered expert testimony meets the requirements of Rule 702 by a preponderance of the evidence. Daubert v. Merrell Dow Pharms., Inc., 509 U.S. 579, 592 & n. 10, 113 S.Ct. 2786, 125 L.Ed.2d 469 (1993) ("Daubert I"). The Rule 702 inquiry requires the Court to determine that the witness is qualified by special knowledge as an expert in the relevant area of expertise. In addition, the Court must scrutinize the proffered expert testimony in order to assure that the expert had sufficient data, used reliable scientific methods, and applied those methods to the data in a reliable way. Finally, the Court must determine if the proffered expert testimony is helpful to the trier of fact. See, e.g., In re Silicone Gel Breast Implants Prods. Liab. Litig., 318 F.Supp.2d 879, 889-93 (C.D.Cal.2004) (dividing the Rule 702 inquiry into three sections analyzing the expert's qualification, the reliability of the testimony, and the usefulness of the testimony). The party proffering the expert testimony bears the burden of establishing by a preponderance of the evidence that the testimony meets the requirements of Rule 702. Daubert I, 509 U.S. at 593 n. 10, 113 S.Ct. 2786 (citation omitted).

Ensuring that an expert's testimony uses reliable data and methodology is often the most challenging step in the Rule 702 inquiry. A district court must be "certain that an expert ... employs in the courtroom the same level of intellectual rigor that characterizes the practice of an expert in the relevant field." Kumho Tire Co. v. Carmichael, 526 U.S. 137, 152, 119 S.Ct. 1167, 143 L.Ed.2d 238 (1999); see also Daubert v. Merrell Dow Pharms., Inc., 43 F.3d 1311, 1316 (9th Cir.1995) ("Daubert II") (stating that Daubert I requires federal judges to "satisfy themselves that scientific evidence meets a certain standard of reliability" beyond "the expert's bald assertion of validity"). To this end, the party proffering the expert testimony "must show that the expert's findings are based on sound science" by providing "objective, independent validation of the expert's methodology." Daubert II, 43 F.3d at 1316. "[T]he test of reliability is flexible" and may include specific factors mentioned in Daubert I and other decisions, but a district court has "broad latitude" in deciding the appropriate factors to consider in determining reliability. Kumho, 526 U.S. at 141-42, 119 S.Ct. 1167. Some factors that the district court may consider are: general acceptance of the method in the scientific community; whether the method has been subject to publication or peer review; whether the known or potential error rate is acceptable; whether the testimony is based on independent research or prepared specifically for litigation; and whether the expert points to an objective source to show that she has followed the scientific method. See Daubert II, 43 F.3d at 1316-19. The circumstances of the particular case will dictate which factors the district court should consider in making its determination. Id. at 150, 119 S.Ct. 1167.

IV. DISCUSSION

Under Rule 702, each of the proffered expert reports describing the statistical sampling methodologies must be authored by a qualified expert witness, reliably apply scientific methods to sufficient data, and be helpful to the trier of fact.

A. Qualifications of the Expert Witnesses

In order to testify as an expert witness, the witness must be "qualified as an expert by knowledge, skill, experience, training, or education." FED. R. EVID. 702. Plaintiffs proffer seven expert reports, six of which are authored by Dr. Charles D. Cowan. The remaining report is authored by Dr. Richard K. Green. Although the Defendants only argue that Dr. Green's qualifications are insufficient under Rule 702, this Court must, in its role as gatekeeper, be satisfied that both Dr. Cowan and Dr. Green are qualified as experts.

1. Dr. Charles D. Cowan, Ph.D.

Dr. Charles D. Cowan, Ph.D. holds a Bachelor of Arts and a Master of Arts in Economics, both from the University of Michigan,1 as well as a doctorate in Mathematical Statistics from George Washington University. Dr. Cowan's experience in the public sector includes serving as the chief statistician for both the FDIC and the National Center for Education Statistics at the U.S. Department of Education. In the private sector, Dr. Cowan served as a director for PricewaterhouseCoopers LLP and consulting firm ARPC before cofounding his own consulting firm, Analytic Focus LLC. Dr. Cowan has designed several economic measurement programs and studies, many of which have included designs for statistical sampling methods. Dr. Cowan has also held several positions in academia and now teaches graduate and undergraduate courses in statistics at the School of Public Health at the University of Alabama.

Dr. Cowan has extensive education and experience directly related to the areas of expertise related to his proffered testimony. Dr. Cowan is therefore qualified to serve as an expert witness in the areas of economics and statistics and is further qualified to design and implement statistical sampling studies.

2. Dr. Richard K. Green, Ph.D.

Dr. Richard K. Green, Ph.D. holds a Bachelor of Arts in Economics from Harvard University as well as a Master of Arts and doctorate in Economics, both from the University of Wisconsin-Madison. Dr. Green's experience in academia is extensive. He currently serves as a business school professor and real estate center director at the University of Southern California. He has served as a dean and professor of real estate, finance, and economics at George Washington University, as a professor of real estate at the Wharton School at University of Pennsylvania, and as a professor at the University of Wisconsin-Madison School of Business. Dr. Green has served as both the Principal Economist and the Director of Financial Strategy and Policy Analysis at Freddie Mac. Dr. Green continues to publish research, scholarly papers, book chapters, and other publications in the areas of economics, real estate, and finance.

The Defendants claim that Dr. Green lacks the requisite level of expertise to provide expert testimony because Dr. Green does not claim to be a statistician. Instead, according to Defendants, Dr. Green only studied statistics during his education in economics, uses statistical analysis in his published articles, and teaches courses that discuss statistical issues. The Defendants' argument is both factually and legally baseless. "Statistical expertise is not confined to those with degrees in statistics. Because statistical reasoning underlies many kinds of empirical research, scholars in a variety of fields — including ... economics ... — are exposed to statistical ideas, with an emphasis on the methods most important to the discipline." David H. Kaye and David A. Freedman, Reference Guide on Statistics, in REFERENCE MANUAL ON SCIENTIFIC EVIDENCE 211, 215 (3d ed. 2011) ("FJC Ref. Guide on Statistics"). A cursory review of Dr. Green's publications shows his familiarity with statistical analysis. His background provides evidence of his exposure to statistical methods in the disciplines of economics and real estate, which are highly relevant to the analysis of RMBS. In addition, "[a] lack of specialization affects the weight of the expert's testimony, not its admissibility." In re Silicone Gel Breast Implants Prods. Liab. Litig., 318 F.Supp.2d at 889. Whether or not Dr. Green claims to be specialized in the field of statistics, as opposed to a field which relies on statistical analysis, does not affect the admissibility of his testimony. Dr. Green, like Dr. Cowan, is qualified as an expert in the areas of expertise relevant to his proffered testimony.

B. Reliability

In order to be admissible expert testimony, each of the proffered expert reports must reliably apply scientific methods to sufficient data. FED. R. EVID. 702(b)-(d). The seven expert reports proffered by Plaintiffs fall into two categories. First, six expert reports2 (collectively, the "Case-Specific Reports") describe a statistical sampling methodology designed to randomly take 100 supporting loans from the SLGs for each of the certificates at issue and re-underwrite each of the 100 supporting loans to determine whether they conform with underwriting guidelines.3 Based on the results of re-underwriting each of the 100 supporting loans selecting in the samples, Plaintiffs intend to show whether Defendants made material misstatements in the Offering Documents regarding, inter alia, the loan-to-value ("LTV") ratio, owner occupancy rates, and compliance with underwriting guidelines.

Second, one expert report4 (the "Broad Report") describes a statistical sampling methodology designed to randomly take 52 loans from each group of loans supporting 20 securitizations (for a total of 1,040 loans) out of the 447 Countrywide RMBS securitizations not at issue in the underlying litigations, reunderwrite the 1,040 loans to determine if each loan meets the underwriting guidelines, and combine the re-underwriting data from the 1,040 loans with the reunderwriting data from the Case-Specific Reports. Based on the results of reunderwriting each of the 1,040 loans selecting in the sample, combined with the reunderwriting of 9,052 loans from the Case-Specific Reports, Plaintiffs intend to estimate the number of non-conforming loans included in all Countrywide RMBS securitizations (as opposed to just those securitizations from which the Plaintiffs purchased) from 2005 through 2007. Plaintiffs believe this will show systematic abandonment of Countrywide's underwriting guidelines during the period in which Plaintiffs purchased certificates in the six cases. Both the Case-Specific Reports and the Broad Report make several explicit assumptions, including the defect rate and the variability of the defect rate, in order to estimate maximum possible margins of error.5

The Case-Specific Reports each describe some, but not all, aspects of the design of the statistical study to be used in the final expert testimony describing the results of the re-underwriting process. The Cowan Allstate Report is exemplary of the design of the studies Dr. Cowan employs in the other five cases. In the Cowan Allstate Report, Dr. Cowan lists a random sample of 100 loans from each of the unique SLGs for each class of certificates purchased by Allstate from the Countrywide RMBS securitizations. In addition, Dr. Cowan lists an additional 100 loans for each class of certificates. Allstate will use the loan files to re-underwrite the first 100 loans. If any loan files are missing from the first 100 loans, a substitute loan will be randomly selected and substituted for the loan with missing loan files.

In an effort to improve the accuracy of the sample, Dr. Cowan will stratify the sample by FICO score. In a statistical sample, the margin of error resulting from extrapolating a small sample onto a large population will be greater when the variability, or spread, of the sample is large. See FJC Ref. Guide on Statistics at 246 ("If the material being sampled is heterogeneous, random error will be large; a larger sample will be needed to offset the heterogeneity."). Stratification by FICO score means that, where the FICO score data is available, the samples will be placed into strata. Each stratum will have a corresponding FICO value or range of FICO values. Every loan will be assigned to one stratum based on its FICO score. If the variability of the loans across each stratum is smaller than the variability of the loans across the sample as a whole, stratification may reduce the overall margin of error for the sample.

Dr. Cowan will then extrapolate the results to draw statistical conclusions about the Countrywide RMBS securitizations from which Allstate purchased certificates. Dr. Cowan states that the sample size will allow him to draw statistical conclusions having a 95 percent confidence level6 with a maximum margin of error of ±10 percent. "The confidence level indicates the percentage of time that intervals from repeated samples would cover the true value." FJC Ref. Guide on Statistics at 247. Dr. Cowan states that the 95 percent confidence level is commonly accepted in scientific fields that use statistical sampling and provides several examples of studies and texts using or recommending the 95 percent confidence level. Dr. Cowan also states that a ± 10 percent margin of error is sufficiently low to draw scientifically valid conclusions and that, due to diminishing returns, analysis of a larger sample population would not provide sufficient benefit in reduction of the margin of error in light of the cost of additional re-underwriting.

Dr. Green's sampling methodology in the Green Hapoalim Report is the same as the sampling methodology described in the Cowan Allstate Report. Dr. Green provides additional detail regarding the statistical software used to generate the random sample, the representativeness testing applied to the sample, and the formulas used to determine the margin of error. In addition, unlike Dr. Cowan's methodology, Dr. Green's statistical analysis will exclude loans with no FICO score. Dr. Green anticipates that the maximum margin of error will be ±10 percentage points for the certificates with larger SLGs and ±7 percentage points for certificates with smaller SLGs.

The Cowan Broad Report details a methodology for providing a statistically valid sample of all loans supporting the Countrywide RMBS between 2005 and 2007, as opposed to a sample of loans underlying one class of certificates. Plaintiffs intend to use the Broad Report to show systematic abandonment of the underwriting standards at Countrywide. Dr. Cowan will randomly select additional loans for re-underwriting ("Broad Sample Loans") from the Countrywide RMBS from which none of the Plaintiffs purchase certificates ("Remaining Countrywide RMBS"). The Broad Sample Loans will be sampled using multi-stage cluster sampling. Dr. Cowan will randomly select a sample of 20 securitizations out of the Remaining Countrywide Securitizations. Next, Dr. Cowan will select 52 loans from each of the 20 securitizations for a total sample of 1,040 loans. Using both these additional 1,040 Broad Sample Loans as well as the 9,052 loans already re-underwritten for analysis in the Case-Specific Reports, Dr. Cowan will use the data from the re-underwriting to estimate an underwriting defect rate for all 447 Countrywide RMBS issued between 2005 and 2007. Dr. Cowan estimates that the maximum margin of error will be ± 10 percentage points.

The Defendants argue that several flaws in the sampling methodology render the expert reports unreliable. The flaws identified by the Defendants fall into two categories: first, deficiencies that affect the statistical methods used in both the Case-Specific Reports and the Broad Report; and second, deficiencies in the statistical methods used only in the Broad Report. The Court addresses one deficiency in the Broad Report sua sponte in its role as gatekeeper. For the reasons set forth in greater detail below, the Court finds that the sampling methodology described in the Case-Specific Reports satisfy the requirements under Rule 702 while the sampling methodology described in the Broad Report fails to satisfy the requirements under Rule 702, thus rendering the Broad Report inadmissible.

1. The Defendants' Objections to the Methodologies of both the Case-Specific Reports and the Broad Report

The Defendants argue that all the proffered expert reports suffer from several common flaws that render the reports inadmissible under subsections (b) and (d) of Rule 702. Both Plaintiffs and the Defendants appear to agree that statistical sampling is a reliable scientific method under Rule 702(c). However, the Defendants argue that the proffered expert reports do not use sufficient data, see FED.R.EVID. 702(b), and do not apply appropriate statistical sampling methods in a reliable way, see FED.R.EVID. 702(d).

a. Objections regarding the sufficiency of the data.

The Defendants offer several arguments that go to the sufficiency of the data analyzed for both the Case-Specific Reports and the Broad Report. The Defendants argue that, first, the use of binary inputs provides inaccurate input data; second, the substitution of missing loan files compromises the representativeness of the input data; and third, the size of the sample for each securitization or group of securitizations is insufficient as shown by the 10 percentage point margin of error.

Use of Binary Inputs. During the re-underwriting process, each re-underwritten loan will be assigned one or more binary inputs for each selected variable, such as loan-to-value ratio, FICO score, income, etc. The binary input will signify either a "yes" or "no" answer to the applicable question. For example, for the question of whether a loan complied with the applicable underwriting guidelines, the binary input would be "yes" if the loan was originated in compliance with the applicable underwriting guidelines or "no" if the loan was not originated in compliance with the applicable underwriting guidelines. According to the Defendants, use of a binary input in the sampling model prevents the model from correctly evaluating questions of gradation, like reasonable professional judgment and materiality. Since the questions relevant to Plaintiffs' claims are gradational, and not binary, use of a binary input would produce inaccurate results.

Although certain questions may not be amenable to binary answers, the experts have adequately posed questions to elicit answers that may properly be classified into binary inputs. Whether a loan complied with an underwriting guideline, whether a property appraisal complied with an applicable standard, or whether a loan-to-value ratio was misstated are all questions that are appropriately answered by an input of "yes" or "no." See, e.g., Assured Guar. Mun. Corp. v. Flagstar Bank, FSB, 920 F.Supp.2d 475, 503 (S.D.N.Y.2013) (finding that the "key determination of `materiality' ... is in fact a binary decision"). The ultimate questions at issue in this litigation will likewise be binary questions: whether the Defendants are liable for making material or fraudulent misrepresentations about the RMBS purchased by Plaintiffs. Where the questions selected are amenable to binary answers and the "fundamental decision [of liability] is a binary one," binary inputs are appropriate and provide sufficient underlying data for analysis. See id. The binary inputs to be used in the statistical samples proposed in the proffered expert reports meet the requirement for sufficient data under Rule 702.

In addition, the Defendants have explicitly reserved their right to object to the reliability of the re-underwriting process. The danger of using binary inputs for questions of gradation is that reasonable people may differ as to the accuracy of the input. If the parties disagree about the accuracy of the binary inputs during the reunderwriting process, the Defendants will have the opportunity to object during or after the re-underwriting process.

Replacement of Missing Loan Files. When the loan tapes for a loan selected as part of the random samples are not available, a new loan from the list of replacement loans will be substituted in its place. The Defendants argue that replacing loans with missing loan tapes could result in a non-random, or nonrepresentative, sample of loans. In particular, a non-random sample could result if loan tapes for a particular group of loans are missing and the data from the group of missing loan tapes is correlated with one or more variables analyzed during re-underwriting. For example, if all the loan tapes for a single originator in Nevada were unavailable and if the unavailable loans had a defect rate different from the average, then substituting in a replacement loan would result in a sample that was not representative of the SLG as a whole.

The Case-Specific Reports describe the methods used by the expert to ensure the representativeness of both the loan sample and the replacement loan sample. Both the loan sample and the replacement loan sample were tested against the SLG population on eleven variables to ensure that the samples accurately represented the SLG population. See, e.g., Cowan Allstate Report ¶ 66. The tests used to ensure representativeness are well-known generally accepted statistical techniques. The Defendants' expert, Dr. Barnett, offers a hypothetical: "Suppose, for instance, that loan files from Nevada are more commonly missing than loan files from other states.... [T]he final samples might thus suffer a geographic bias." Rebuttal Report of Arnold Barnett, Ph.D. A hypothetical suggestion, with no supporting facts and no reason to infer that such a suggestion might in fact be true, cannot alone be enough to overcome facts that support the scientific reliability of the sample. See Assured Guar. Mun. Corp., 920 F.Supp.2d at 503 ("[W]hile the burden is on Assured [to show a sampling methodology is reliable], Flagstar presented no evidence [] suggesting that the samples are not representative as to these particular variables. [T]he Court concludes that the samples are duly representative."). The loan tape replacement methodology in the Case-Specific and Broad Reports reliably applies accepted statistical methods.

Use of an Inadequate Sample Size in the Case-Specific Reports. In order to draw reliable conclusions about a population based on a statistical sample, the sample size must be large enough to support those conclusions. In statistics, the margin of error is used to express the amount of error that results from using a sample smaller than the whole population to draw conclusions about the whole. The Case-Specific Reports each use a loan sample of 100 loans to produce a confidence level of 95 percent with a maximum margin of error of ±10 percentage points.7 The margin of error depends at least in part on the relationship between the size of the sample and the size of the population. See FJC Ref. Guide on Statistics at 243 (Generally, "the standard error for the sample average can be computed from (1) the size of the sample ... and (2) the standard deviation of the sample values.... Bigger samples give estimates that are more precise. Accordingly, the standard error should go down as the sample size grows, although the rate of improvement slows as the sample gets bigger.") For small samples in large populations, the size of the sample and the desired confidence level will determine the number of items in the sample. In the Case-Specific Reports, a sample size of 100 loans was selected in order to obtain the 95 percent confidence level and the ±10 percentage point maximum margin of error. The Defendants argue that the maximum margin of error is too high and that the expert reports fail to show that such a high margin of error is considered reliable in the field. This argument is essentially equivalent to the argument that the loan sample size of 100 loans is insufficient, since it is the loan sample size that determines the estimated maximum margin of error. Id. at 246 ("Generally, increasing the size of the sample will reduce the level of random error.") (parenthesis omitted).

The parties and their experts provide numerous examples of the use of statistical sampling to support their respective positions regarding the acceptability of a ±10 percent margin of error. The majority of examples cited by the parties, however, are not relevant to the issue of whether or not the margin of error is considered reliable in scientific fields. Dr. Cowan points to at least two examples of scientific studies that used a margin of error of ±10 percentage points: a government study of U.S. Food and Drug Administration management techniques, 2009 FDA Managers Survey on Performance and Management Issues, GAO-10-280SP, February 2010, an e-supplement to GAO-10-279 (available at http://www.gao.gov/special.pubs/gao-10-280sp/), and a research study on vegetation, Birdsall, et al., Image Analysis of Leafy Spurge (Euphorbia esula) Cover, 11 Weed Sci. Society of America 801 (No. 4 Oct.-Dec. 1997). Dr. Green points to several examples of academic literature on statistics that describe the use of a 100 item sample size to extrapolate results to a large population. The examples provided by the experts are sufficient to show that a ± 10 percentage point margin of error is a reliable application of statistical methods and that a 100 item sample size comprises sufficient data for a sample of a large population.

In response, the Defendants argue that many studies use a smaller margin of error and that a smaller margin of error offers greater precision. Defendants cite, for example, the U.S. Department of Housing and Urban Development's sampling standards requiring a ±2 percent margin of error for its own residential mortgage loan insurance and acquisition program. However, the U.S. Department of Housing and Urban Development's sampling requirements for its own purposes offer no insight into the scientific validity of a ±10 percentage point margin of error. The government can choose whatever level of precision it deems desirable for instituting its own regulations. The quality control requirements instituted by the U.S. government are not scientific studies, but policy choices.

The Defendants offer other similarly true but irrelevant arguments. The Defendants highlight at length the reports proffered by Dr. Cowan in other RMBS litigation in which the margin of error was lower than those proffered in the present litigation. These litigation positions are not relevant to the issue here: whether or not a ±10 percentage point margin of error is a reliable application of statistical sampling. The Defendants' further arguments regarding the cost-benefit analysis for sample size are also irrelevant. As this Court has made clear, Plaintiffs may choose how to present their own case. Litigation is expensive, and each party has the discretion to decide where valuable litigation dollars are best spent. The Defendants offer no scientific evidence to contradict the scientific studies cited by Plaintiffs' experts that indicate a 95 percent confidence level with a ±10 percentage point maximum margin of error comports with scientific standards.

The Defendants also offer a related argument — the samples are not large enough to provide reliable results with regard to specific originators. The Court agrees that the sampling methodology, as currently designed, does not contain sufficient data to show defect rates regarding specific originators. To the extent that the Defendants argue that proof as to a defect rate is required for each originator, the argument is one of helpfulness, which will be addressed below.

b. Objections regarding reliable application of the scientific method.

The Defendants offer several arguments that go to the reliable application of the statistical sampling methodology for both the Case-Specific Reports and the Broad Report. The Defendants argue that Plaintiffs have failed to provide sufficient evidence showing the experts used reliable statistical sampling methods, that stratification by FICO score is a reliable application of the stratification technique, and that reliable statistical sampling methods require the experts to propose an extrapolation technique at the time the sampling method is proposed.

Lack of Independent Research, Peer Review, or an Objective Source. In arguing that Plaintiffs fail to show that the experts used reliable statistical sampling methods indicated in the reports, the Defendants offer an aggressive interpretation of the Court's gatekeeping requirement under Daubert II. According to the Defendants, Daubert II requires that the proposed sampling methodology be "recognized [by a] minority of experts designing similar sampling protocols for loan re-underwriting outside of the litigation process." Countrywide Defendants' Reply Memorandum of Points and Authorities in Further Support of their Motion to Exclude at 21, Allstate Insurance Co. v. Countrywide Financial Corp., Case No. 2:11-cv-5236-MRP-MAN (Doc. No. 290). That interpretation of Daubert II is misleading. Instead, Daubert II requires that the Court assure itself that, at each point in the process, the experts have properly used accepted statistical techniques and formulas to achieve a reliable result. See Daubert II, 43 F.3d at 1317-19. Since the sampling methodology grew out of litigation and has not been the subject of normal scientific scrutiny, the experts may point to objective sources to support the proposed methodology. Id. at 1319.

The Case-Specific Reports and the Broad Report point to reference texts and academic literature that support the proposed sampling methodology. "Because most statistical methods relied on in court are described in textbooks or journal articles and are capable of producing useful results when properly applied, these methods generally satisfy important aspects of the `scientific knowledge' requirement in Daubert [I]." FJC Ref. Guide on Statistics at 214. These methods are equally able to satisfy the objective source requirement of Daubert II. Here, Plaintiffs and their experts have pointed to textbook methods and academic literature supporting the techniques and formulas used in their sample design and margin of error calculations. Therefore, the Case-Specific Reports and Broad Report have properly supported the sampling methodology by pointing to objective sources under Daubert II.

Non-Ideal Use of Stratification. The Defendants also argue that stratification by FICO score is unreliable as used in the Case-Specific Reports. The Plaintiffs' experts explain that stratification is a statistical technique that can increase the precision of estimates obtained by sampling. Stratification divides a "population into relatively homogeneous groups" called strata, draws a sample from each group, and weights the results from each group to extrapolate a conclusion. FJC Ref. Guide on Statistics at 299. Although stratification does not guarantee any reduction in the margin of error, it can under certain circumstances produce a smaller margin of error. Dr. Cowan provides data from a statistics textbook describing the use and benefits of stratification. According to Defendants, although stratification is an acceptable statistical technique, Dr. Cowan and Dr. Green put forth no evidence to suggest that stratification by FICO score will lower the margin of error in this case. Instead, Defendants argue that stratification by another variable, or more than one variable, could provide more reliable results than stratification by FICO score.

In making this argument, Defendants misapprehend the purpose of the Daubert reliability analysis. The Daubert standard does not exist to ensure that only the most ideal scientific evidence is admissible in court proceedings, but instead to ensure that expert testimony is "derived by the scientific method." Daubert I, 509 U.S. at 590, 113 S.Ct. 2786; see also Deutsch v. Novartis Pharms. Corp., 768 F.Supp.2d 420, 431 (E.D.N.Y.2011) ("Under Daubert, an expert need not base his opinion on the best possible evidence, but upon `good grounds, based on what is known.'") (citation omitted). Stratification cannot increase the margin of error, although it can increase the complexity of the analysis. The Plaintiffs' experts estimate a maximum margin of error given the proposed sampling methodology, which necessarily assumes the worst-case scenario that stratification did not help, and also did not hurt, the margin of error. In addition, the Plaintiffs' experts explain that stratification by additional variables may not decrease the margin of error and are subject to diminishing returns as more variables are used for stratification. Given this information, it is logical for an expert to limit the variables for stratification, as Plaintiffs' experts have done.

The use of FICO scores as the selected stratification variable comports with common sense; there is reason to believe defect rates could be higher for loan files with a lower FICO score since higher FICO scores indicate a positive borrower credit history and a lower risk borrower profile. For borrowers with a lower FICO score, there plausibly could be greater incentive to misrepresent other variables on the loan application, resulting in a higher defect rate. But even if this assumption is incorrect, stratification by FICO score cannot increase unreliability of the results. Perhaps the Defendants are correct, and stratification by another variable would decrease the margin of error more than stratification by FICO score. The possibility of a more reliable result, however, does not decrease the reliability of the proposed sampling methodology. The Court's gate-keeping task is not so strict as to allow only the best possible scientific evidence, but to allow only reliable scientific evidence. Stratification is a widely accepted statistical technique, and using FICO score as a stratifying variable is a reliable application of that technique that may increase the reliability of the results of the sample.

Lack of an Extrapolation Method. The Defendants argue that the expert reports are not scientifically reliable because the expert reports do not set forth in advance the methodology for extrapolating the results of the statistical sampling to the population as a whole. The Defendants' rebuttal expert explains that the reliability of a sampling method cannot be fully assessed without information about the extrapolation technique.

The expert reports in their current state will not be used as admissible evidence for any substantive motions or trial proceedings. A sampling methodology without a result is obviously not adequate to prove any result. Since Plaintiffs have chosen not to propose an extrapolation methodology at this point in the re-underwriting process, the Court cannot rule regarding the reliability of the extrapolation methodology.

To the extent that the Defendants use this argument to imply that it is not scientifically acceptable to select a method of extrapolation after data are obtained, the Court does not agree. Neither party has presented any expert testimony regarding the timing for selecting the extrapolation method. The Defendants instead compare Dr. Cowan's decision to select the extrapolation method later to Dr. Cowan's timing for selecting the extrapolation method in other litigations. Once again, the Defendants confuse the issue of scientific validity and litigation tactics. Dr. Cowan's decision not to provide an extrapolation methodology concurrently with the sampling methodology appears to be a litigation decision unrelated to scientific validity. Without any scientific evidence on the proper timing for selection of the extrapolation method, the proper time to decide on the admissibility of the extrapolation methodology will be when expert testimony including the extrapolation of the data has been submitted.

2. The Defendants' Objections to the Methodology of the Broad Report

The Defendants offer two arguments that address deficiencies found only in the Broad Report. The Defendants argue that, first, Dr. Cowan incorrectly estimated the margin of error for the Broad Report, and second, the multi-stage cluster sampling is functionally equivalent to an analysis of the specific securities Plaintiffs purchased and not the Countrywide RMBS issued from 2005 to 2007 as a whole.

Inaccurate Margin of Error. The Defendants' rebuttal expert, Dr. Barnett, provides an example of a defect rate distribution using a statistical simulation that results in a margin of error of ±22 percentage points for the sampling methodology used in the Broad Report, far greater than the ±10 percentage point maximum margin of error. Dr. Barnett also asserts that Dr. Cowan's selected formula for calculating margin of error weights securitizations equally across the sample regardless of actual loan population size and that it uses a formula for variance that does not appear in the objective source cited by Dr. Cowan.

In response, Dr. Cowan explains the desirability of using several alternative extrapolation methods and selecting the method with the lowest margin of error. Plaintiffs provide page 305 of William G. Cochran's Sampling Techniques, one of the objective sources cited by Dr. Cowan. The textbook shows the variance formula used in Dr. Cowan's selected extrapolation method, and the textbook heading notes that the example on page 305 falls under the chapter for subsampling with units of unequal size. Dr. Cowan provides his calculation using the third method, including values for each variable in the equation. By providing a copy of the textbook and showing his assumptions and calculations, Dr. Cowan connects his selected methodology and application to an objective source. Accordingly, Dr. Cowan's calculation of the maximum margin of error meets the admissibility requirement for scientific reliability.

Unreliable Application of Multi-Stage Cluster Sampling. The Defendants argue that the Broad Report's multi-stage cluster sampling amounts merely to an analysis of what the Plaintiffs bought. The crux of this argument appears to be twofold. The multi-stage cluster sampling is unreliable because, first, the sample size is too small8 and second, the multi-stage cluster sampling technique has not been reliably applied to the two strata of data.

With regard to the sample size, the purpose of using a sample is to extrapolate results from a small sample to a large population. The adequacy of the sample size is, at least in part, reflected in the margin of error calculation. Since the Court has considered and discarded the Defendants' more specific arguments relating to the margin of error, the Court sees no reason to entertain this more general criticism.

With regard to the application of the multi-stage cluster sampling technique, a "multistage cluster sample" is "[a] probability sample drawn in stages, usually after stratification; the last stage will involve drawing a cluster." FJC Ref. Guide on Statistics at 290. The "cluster sample" provides a random sample by making a random selection and sampling within a cluster around the random selection. Id. at 284. Dr. Cowan's cluster sampling is based on the selection of certain certifications within the Remaining Countrywide RMBS; he then selects clusters of loans from each randomly selected securitization. The "multi-stage" aspect of the cluster sampling results from the combination of the two superstrata, the sample loans from Plaintiffs' certificates and the sample loans from the Remaining Countrywide RMBS.

Dr. Cowan compares his use of the multi-stage cluster sampling technique to the methodology used by the state of California to determine the unemployment rate. However, the analogy between the two applications of the multi-cluster sampling technique is inapposite. The California unemployment analysis uses a two stage sampling technique to create a first superstratum composed of samples from each major county that makes up 95 percent of California's total population. The remaining counties are then represented by a second superstratum composed of a subset of households selected from all the remaining counties. As the Defendants argue, Dr. Cowan does the opposite. In his multi-stage cluster sampling technique, Dr. Cowan uses a large sample from a smaller total population, the loans from Plaintiffs' certificates, and then offsets the remaining majority of securitizations with a subset of loans from the larger superstratum, which includes all Remaining Countrywide RMBS. Dr. Cowan presents no scientific evidence or academic literature indicating that his application of the multi-stage cluster sampling technique is a reliable application of the technique. The unreliability of the proposed multi-stage cluster sampling technique renders the Broad Report inadmissible under Rule 702(d). FJC Ref. Guide on Statistics at 214 ("Of course, a particular study may use a method that is entirely appropriate but that is so poorly executed that it should be inadmissible under Federal Rules of Evidence 403 and 702.") Plaintiffs fail to show that Dr. Cowan's multi-stage cluster sampling methodology is a reliable application of the accepted multi-stage cluster sampling technique.

3. The Court's Objection to the Methodology of the Broad Report

Although the unreliable application of the multi-stage cluster sampling technique renders the Broad Report inadmissible under Rule 702, the Court also finds that, as an alternative basis for exclusion, the Broad Report is unreliable due to the potential for systematic error.

For any statistical sample, the accuracy of the results depends on two factors, random error and systematic error. FJC Ref. Guide on Statistics at 296. Here, random error is shown as the maximum margin of error calculations in both the Case-Specific Reports and the Broad Report. Id. at 240. The margin of error calculations account for the variance and sample size. Id. at 243. Indeed, much of the Defendants' argument attempts to show random error by critiquing the size of the proposed sample.

Systematic error, or bias, on the other hand, cannot be reduced by increasing sample size. FJC Ref. Guide on Statistics at 246. Bias creates a "systematic tendency for an estimate to be too high or too low." Id. at 283; see also FAIGMAN ET AL., 1 MODERN SCIENTIFIC EVIDENCE § 1.22 (2012-13 ed.) ("Systematic errors, unlike random errors, tend to work in a single direction and, therefore, introduce bias into the data."). Although Plaintiffs' experts provide a maximum margin of error for the sampling methodology, the margin of error calculation fails to account for potential systematic errors. See FJC Ref. Guide on Statistics at 249 ("Standard errors and confidence intervals generally ignore systematic errors such as selection bias or nonresponse bias.").

The Court's primary concern with the proposed sampling methodology for the Broad Report is the potential for systematic error in the form of selection bias. The Broad Report's sample is comprised of a comparatively high percentage of loans (89.7 percent or 9,052 loans) selected from Plaintiffs' certificates at issue in the underlying litigations and a comparatively low percentage of loans (10.3 percent or 1,040 loans) selected from certificates not at issue in the underlying litigations. The Broad Sample's substantial reliance on loans supporting certificates selected for litigation strikes the Court as a clear example of selection bias, or "nonrandom selection of subjects for study." FJC Ref. Guide on Statistics at 296. Indeed, "[s]election bias is acute when ... attorneys choose cases for trial." Id. at 225. More specifically, Plaintiffs introduce a potential form of selection bias in the Broad Sample by re-underwriting a significant portion of loans supporting certificates pre-selected for litigation.9 Each of these certificates has been subjected to various pre-litigation activity, likely including investigation revealing the strength of the evidence, the value of the suit, and the likelihood that wrongdoing occurred. Based on these pre-litigation efforts, Plaintiffs are more likely to have filed suit on certificates yielding the greatest litigation advantage, which in turn may affect the ability of the Broad Report's sample to represent a population that includes many loans that have not been selected for litigation.

The Court cannot countenance the use of this type of convenience sample that is "easy to take but may suffer from serious bias." FJC Ref. Guide on Statistics at 285; see also In re Bextra and Celebrex Mktg. Sales Practices and Prod. Liab. Litig., 524 F.Supp.2d 1166, 1176 (N.D.Cal. 2007) (rejecting expert testimony that "cherry-pick[ed]" studies to analyze in support of the expert's conclusion); cf. Mark Haug and Emily Baird, Finding the Error in Daubert, 62 Hastings L.J. 737, 739 (2011) (suggesting that an expert must account for both random error and bias before being permitted to testify under Daubert). Other courts have also rejected convenience samples suffering from litigation selection bias. One example arose in In re Chevron U.S.A., Inc., 109 F.3d 1016 (5th Cir.1997). According to the FJC Reference Guide on Statistics:

FJC Ref. Guide on Statistics at 225 n. 32. Each RMBS certificate and loan has different sets of representations, insurance, underlying underwriting standards, and other varying factors. It is highly likely that parties holding non-representative certificates would be the most motivated and persistent litigants, making the Case-Specific Report samples resemble the "extreme cases" in Chevron. It is exactly that problem that makes these certificates most dangerous for use as a representative sample of the entire universe of Countrywide RMBS.

Without a random sample for the Broad Report, there is no reliable way to draw conclusions about the relationship between Plaintiffs' certificates and Countrywide RMBS as a whole. Experimentation in the Law: Report of the Federal Judicial Center Advisory Committee on Experimentation in the Law 18 (Federal Judicial Center 1981) ("[W]ithout randomization there are no certain methods for determining that observed differences between groups are not related to the preexisting, systematic difference."). If Plaintiffs' certificates systematically differ from the entire group of Countrywide RMBS, then the Broad Report will also differ systematically from the entire group. Stated differently, assuming the methodology in the Broad Report, if applied, estimates a high rate of nonconforming loans, it would be impossible to determine whether, and to what extent, that rate is the result of systematic derogation of underwriting guidelines or strategically selected certificates for litigation.

The Court finds that the Broad Report is inadmissible under Rule 702 because the underlying data is insufficient and the sampling methodology cannot be reliably applied due to systematic error resulting from use of the litigated loans as the primary source of underlying data.

C. Helpfulness

Finally, in order to be admissible under Rule 702, "the expert's scientific, technical, or other specialized knowledge will help the trier of fact to understand the evidence or to determine a fact in issue." FED.R.EVID. 702(a). The Case-Specific Reports indicate that the sampling methodology will be used with the results of the re-underwriting "to establish" several material facts, including, for example, "that Defendants' statements in the Offering Materials were false regarding (i) whether the mortgage loans were originated in compliance with applicable underwriting guidelines; (ii) whether the mortgaged properties were accurately appraised in compliance with applicable appraisal standards; (iii) the number/percentage of loans that had LTV and/or CLTV ratios above specified values; and (iv) the number/percentage of loans that were collateralized by properties that were owner-occupied." Cowan Allstate Report ¶ 28.

The Defendants argue that the high maximum margin of error prevents the results of the sample from being helpful to the trier of fact. At this time, the results of the re-underwriting and the sample extrapolation are not available. The sampling methodology alone does not provide sufficient information for the Court to determine whether or not the final results of the study will be helpful to the trier of fact. Although "[w]hen dealing with challenges to the reliability of scientific evidence, the court must focus on the methodology ... used to draw a conclusion and not the conclusion itself," the helpfulness of the scientific evidence requires an assessment of the conclusion itself. See Deutsch, 768 F.Supp.2d at 425, 453. Other courts have found that conclusions from RMBS sampling with a similar margin of error may be helpful to the jury. See, e.g., Fed. Hous. Fin. Agency v. JPMorgan Chase & Co., 2012 WL 6000885, *9, Case No. 11-cv-6188-DLC (S.D.N.Y. Dec. 3, 2012) (holding that whether or not RMBS sampling with a margin of error of ±10 percentage points permitted a jury to make a finding as to the falsehood of certificate specific statements did not go to Daubert relevance but instead to the weight of the evidence). Without results to evaluate for potential helpfulness, the Court cannot determine if the results of the sample will be helpful to the trier of fact.

The Defendants also argue that due to the small sample size, no conclusions may be reliably drawn regarding the defect rates for specific originators. Without evidence about specific originators' defect rates, the Defendants assert that the Plaintiffs cannot prove their case. The Court agrees, as discussed above, that the sample sizes are insufficient to drawn reliable scientific conclusions regarding the defect rates for individual originators. As to the assertion regarding Plaintiffs' ability to prove their cases, the legal argument is better addressed in a motion on the merits, not on a motion to exclude expert testimony. The legal elements of Plaintiffs' claims and the evidence required to support those claims has not been fully addressed in this procedural motion.

The Court also recognizes the Defendants' concern about Plaintiffs' intent to use the Case-Specific Reports to show compliance rates with "applicable" underwriting and appraisal standards. The expert reports do not indicate which underwriting guidelines are used in the re-underwriting. The final expert reports must include that information. Different underwriting guideline may be relevant to different claims, for example, if a claim against Countrywide pertains to its role as an originator or issuer. However, at this time, without information regarding which underwriting guidelines will be used, the Court again cannot determine whether or not the conclusions from the Case-Specific Reports will be helpful in determining a fact at issue.

The Defendants also argue that the Broad Report is insufficiently designed to show systematic abandonment. Any argument seeking to exclude the Broad Report on the grounds that it would not be helpful to the trier or fact is moot in light of this Court's finding that the Broad Report's sampling methodology is not scientifically reliable. However, to address the Defendants' point about the "aggregate" or average defect rate, the Court again finds that the sufficiency of Plaintiffs' evidence to prove any elements of their claims is more appropriately addressed in a motion on the merits.

V. CONCLUSION

The Court GRANTS IN PART and DENIES IN PART the Defendants' Motion to Exclude the Proffered Expert Reports of Charles M. Cowan and Richard K. Green. The Court finds that Charles M. Cowan and Richard K. Green are qualified to serve as experts in statistical sampling methodology. The Court finds that the sampling methodology described in the Broad Report shall be excluded as scientifically unreliable under FED. R. EVID. 702(b)-(d). The Court finds that the sampling methodology described in the Case-Specific Reports meets the requirements for scientific reliability under FED. R. EVID. 702(b)-(d). The Court will not entertain any further Daubert motions regarding loan sampling in the above-captioned cases unless brought in conjunction with a motion for summary judgment.

IT IS SO ORDERED.