ROBERT C. CHAMBERS, District Judge.
Pending before the Court is Plaintiffs' Motion for Relief Related to Ford's Discovery Misconduct. ECF No. 995. In their motion, Plaintiffs argue this Court should sanction Defendant Ford Motor Company for conduct related to the production and protection of source code during discovery. The matter being fully briefed, the Court
Since January 2015, the parties have extensively litigated the scope and manner of source code production by Ford. Throughout the process, Ford has attempted to encumber and restrict Plaintiffs' access to the source code for a number of reasons including, but not limited to: (1) Ford's unwavering assertions its source code is a highly proprietary trade secret that is vigilantly safeguarded at Ford's facilities; (2) some of the source code sought by Plaintiffs is unnecessary for them to test their theory of their case; and (3) Plaintiffs' requests are extraordinarily burdensome and cost prohibitive. The parties' disagreements resulted in many meet-and-confer conferences, numerous motions, multiple hearings, and hundreds of hours spent by both the parties and the Court tackling these issues.
As a result of Ford's concerns and other representations it made about the nature of the source code itself, Plaintiffs assert Ford was able to get a series of onerous and restrictive protective orders which limited the scope of the source code disclosed, required the source code to be in read-only format, and restricted Plaintiffs' experts' review of the source code to a secure room at a Ford facility in Dearborn, Michigan. Plaintiffs contend these conditions were arduous and costly, and impaired their ability to perform testing. Plaintiffs further claim that they now can demonstrate these draconian measures were largely unnecessary based upon Ford's prior conduct. Plaintiffs discovered that Ford used unsecured networks to email third parties sets of source code in a format that could be read, altered, and manipulated. Plaintiffs argue these emails prove Ford misrepresented the level of security and restrictions it placed on the source code and reveal Ford did not provide Plaintiffs with all the source code that the Court ordered it to produce.
In light of these emails, Plaintiffs insist Ford intentionally misdirected the Court and obstructed the discovery process, resulting in a colossal waste of time and money. To remedy this alleged recalcitrance, Plaintiffs ask this Court to enter an order accepting as fact Plaintiffs' theory of the case, prohibiting Ford from relying upon or introducing the non-disclosed source code, and awarding attorneys' fees and costs. Ford disputes Plaintiffs' claims and argues sanctions are unwarranted.
In support of these remedies, Plaintiffs ask this Court to use its power under Rule 37 of the Federal Rules of Civil Procedure and its inherent authority to issue sanctions. Rule 37(b)(2) of the Federal Rule of Civil Procedure permits a court to sanction a party for failing to comply with discovery orders. See Fed. R. Civ. P. 37(b)(2) (stating that a court "may issue further just orders" when a party "fails to obey an order to provide or permit discovery"). In addition, Rule 37(b)(2)(A)(i), provides a court may "direct[] that the matters embraced in the order or other designated facts be taken as established for purposes of the action, as the prevailing party claims[.]" Fed. R. Civ. P. 37(b)(2)(A)(i). Pursuant to subsection (ii), a court further may "prohibit[] the disobedient party from supporting or opposing designated claims or defenses, or from introducing designated matters in evidence[.]" Fed. R. Civ. P. 37(b)(2)(A)(ii).
Without doubt, this Court also "possess[es] certain `inherent powers,' not conferred by rule or statute, `to manage . . . [its] own affairs so as to achieve the orderly and expeditious disposition of cases.'" Goodyear Tire & Rubber Co. v. Haeger, 137 S.Ct. 1178, 1186 (2017) (quoting Link v. Wabash R. Co., 370 U.S. 626, 630-31 (1962)). This power includes the authority "`to fashion an appropriate sanction for conduct which abuses the judicial process.'" Id. (quoting Chambers v. NASCO, Inc., 501 U.S. 32, 44-45 (1991)); see also Felman Prods., Inc. v. Indus. Risk Ins., No. 3:09-0481, 2011 WL 4547012, at *8 (S.D. W. Va. Sept. 29, 2011) (stating a court has "broad inherent power to protect the administration of justice by levying sanctions in response to abusive litigation practices" (internal citations omitted)). Although a court must exercise its inherent powers "with restraint and discretion," a court may look to its inherent power "to fill in the interstices" not covered by the Rules or a statute. Chambers v. NASCO, Inc., 501 U.S. 32, 44 & 46 (1991) (citation omitted). If misconduct can be adequately addressed under the Rules, "the court ordinarily should rely on the Rules rather than the inherent power. But if in the informed discretion of the court, neither the statute nor the Rules are up to the task, the court may safely rely on its inherent power." Id. at 50. For instance, the Fourth Circuit stated that Chambers authorizes a court to use its inherent power to impose sanctions "when a party deceives a court or abuses the process at a level that is utterly inconsistent with the orderly administration of justice or undermines the integrity of the [judicial] process[.]" United States v. Shaffer Equip. Co., 11 F.3d 450, 462 (4th Cir. 1993).
Against these precepts, the Court turns to the issues at hand: (1) first, whether Ford made material misrepresentations to the Court that impacted this litigation to the detriment of Plaintiffs; (2) second, whether Ford failed to produce source code in violation of the Court's orders; and (3) third, if the answer to either of the preceding two questions is yes, what sanction, if any, should be imposed against Ford. Additionally, if the Court finds sanctions are warranted under either the Rules or its inherent authority, it must be cautious to impose sanctions that are proportionate to the gravity of the misconduct. In re Jemsek Clinic, P.A., 850 F.3d 150, 158 (4th Cir. 2017), reh'g denied (Mar. 31, 2017) (citation omitted).
Ford argues Plaintiffs have shown no evidence through their motion that warrants sanctions. However, the Court disagrees. Ford has claimed throughout this process that the source code for its Electronic Throttle Control ("ETC") system "has never been produced or provided to any third party, even a Ford supplier, in the manner that Plaintiffs propose, and for good reason." Ford Motor Co.'s Mem. Regarding Source Code Protective Order, at 2 (May 22, 2015), ECF No. 502. Ford stated it would be catastrophic if the source code was divulged to its competitors or terrorists. Id. Ford insisted that, if it was necessary for someone on the outside to see the source code for some purpose, it had to be viewed "`at a Ford facility on a Ford supplied computer with a Ford employee present and operating the computer.'" Id. at 9, quoting Decl. of Eric Luehrsen, at 5, ¶21 (May 22, 2015), ECF No. 502-1.
Following several meet and confer discussions and briefing, the Honorable Cheryl A. Eifert, Magistrate Judge, held a hearing on the ETC source code discovery on May 28, 2015. At the hearing, there was a lengthy discussion and expert testimony about the format and testing of the source code that should be permitted. Due to its concerns, Ford was adamant the source code only should be produced in read-only format, rather than write-access format as sought by Plaintiffs. Ford argued in great detail that write-access format was unnecessary for Plaintiffs to test their theory of their case
At the hearing, Ford also zealously maintained it never permitted the source code to leave its possession. Ford stated its "ETC source code has never been produced, ever, never, ever been produced in the format that the plaintiffs are requesting to anyone. Not suppliers, not third-party suppliers, no one." Id. at 13. Magistrate Judge Eifert specifically asked defense counsel whether Ford had ever used a virtual private network (VPN) or allowed the code to be reviewed anywhere other than a secure Ford facility. Id. at 115. Defense counsel represented that it had not occurred, and its restrictions apply to everyone, including Ford's employees. Counsel further represented that a VPN was not secure enough to alleviate Ford's concerns about keeping the source code protected. Id.
As a result of Ford's representations that calamity would strike if the source code ever left Ford's facilities, the Magistrate Judge issued an Order on June 12, 2015, limiting Plaintiffs' review of the code to a read-only format. Mem. Op. and Order, at 3 (June 12, 2015), ECF No. 543. The Magistrate Judge stated that, "[u]nder the proportionality analysis, at this juncture, any additional benefit of providing Plaintiffs with the code in a write-access format does not outweigh Ford's concerns about the security of its highly proprietary ETC source code." Id., at 17 (footnote omitted). In addition, the Court rejected Plaintiffs' suggestion of using a VPN. Instead, the Magistrate Judge held that the source code must be reviewed "in a secure room in a secure facility where access to the room may be strictly controlled." Id. at 21 (footnote omitted). Ultimately, the secure room was set up at a Ford facility in Dearborn, Michigan.
Despite Ford's steadfast assertions its source code was assiduously protected, the Court finds it is now undeniable that source code was emailed to third parties in unprotected formats. When Plaintiffs' expert, Dr. Philip Koopman, was reviewing documents provided by Ford, he noticed a complete "bookshelf file" for the Powertrain Control Module ("PCM") Main Central Processing Unit (CPU) attached to an email. A "bookshelf file" is code that is in its final or near final form. In his Declaration, Dr. Koopman said he found "over two dozen complete PCM bookshelf files that had not been produced in discovery materials and hundreds of sections of code that Ford had circulated internally and externally." Decl. of Philip Koopman, Ph.D., at 2, ¶6 (June 16, 2017), ECF No. 998. Dr. Koopman stated some of the source code was sent to third parties, including to other manufacturers, such as Mazda and Aston Martin, and to suppliers, such as Roush and Melco. Id. at ¶¶8-10. In addition, Dr. Koopman found many emails indicated "an explicit intent and willingness to send or receive bookshelf file materials with external companies, and one email specifically mentioned a concern about attachments making it past Ford's firewall." Id. at ¶11.
In their briefing, Plaintiffs also give what they refer to as other "representative examples" of source code Ford emailed to third parties. Pls.' Mem. of Law in Supp. of their Mot. for Relief Related to Ford's Disc. Misconduct, at 11 n.18 (June 16, 2017), ECF No. 997. As their first example, Plaintiffs state a Ford employee emailed a Mazda employee a bookshelf copy of "ETC C Code" and a calibration file for a particular vehicle. Plaintiffs assert the file contains content that Ford refused to give them in the secure room. Additionally, Plaintiffs claim it appears that Ford permitted the unprotected "ETC C Code" to be shared with another outside supplier. A year later, the same Ford employee emailed an unencrypted complete bookshelf file for the Ford Fusion's ETC main CPU.
Similarly, Plaintiffs assert they discovered emails in which Ford sent bookshelf files of the ETC Code for the two and three track pedal system to Roush, an international supplier of ETC systems. Plaintiffs claim Ford essentially sent all the .c source code to Roush. Likewise, Plaintiffs claim Ford sent Aston Martin an unencrypted complete set of ETC "raw code." Plaintiffs claim that the .c source code sent to these third parties was not in a read-only format, and it could be put into a complier with minimal effort.
In light of these emails, Ford does not deny that source code was disseminated to third-parties outside of its facility; nor could it. Instead, Ford now attempts to distinguish the source code in the secure room from the source code attached to the emails by claiming the emails did not contain complete sets of source code,
To the contrary, Dr. Koopman states in his Affidavit there was "no read-only access control in place for the plain text source code files that Ford e-mailed around the world," and Ford's distinction between native and flat file source code is nonsensical. Aff. of Philip Koopman, Ph.D., at 4-5, ¶¶10 & 12 (Oct. 19, 2017), Exhibit A attached to ECF No. 1041, at 27-28.
Based upon this Court's review, the Court agrees with Dr. Koopman that Ford's attempt to draw a distinction between native and flat file source code is nonsensical with respect to confidentiality concerns. The bookshelf files attached to the emails are written in plain ASCII text, which are no different than other text files. As such, "[a] normal text editor such as Notepad can read, modify, and write any line of code in the bookshelf file. It is not in `read only' format in any manner whatsoever." Id. at ¶20. In light of this evidence, the Court finds Ford clearly misrepresented the level of security it provided to its own source code. In fact, it appears that, quite the opposite of Ford's claims, editable source code was emailed over unsecured networks by Ford employees to third parties on numerous occasions. It is certainly untrue that Ford only permitted third parties to view source code at a secure Ford facility.
Although Ford now admits third parties were sent source code, it further attempts to justify its actions by explaining that the third parties who received the source code were working with Ford and needed the code to perform their jobs. Ford argues these contractors were bound by the same confidentiality provisions as Ford employees and, therefore, presumably the source code was protected. While it may be true that the third parties who received the source code were bound by confidentiality provisions similar to Ford employees, this argument completely ignores the fact that Plaintiffs' counsel and experts also are bound by strict confidentiality provisions and protective orders. In fact, in hindsight, it appears that the confidentiality provisions imposed upon Plaintiffs and their experts are much more exacting than what Ford employees were doing in actual practice. Furthermore, one of Ford's primary arguments to support its contention about why it was necessary for the source code to be restricted to an ultra-secure Ford facility was because of its fear that an outside computer system could be hacked. Obviously, a third-parties' computer system can be hacked regardless of whether there are confidentiality agreements.
Based upon these facts, the Court has no difficulty determining that Ford blatantly misrepresented the extent to which it both safeguarded its source code at its facilities and suggested the difficulty of producing the source code in discovery. Clearly, it did not require everyone who needed to see the source code to view it "`at a Ford facility on a Ford supplied computer with a Ford employee present and operating the computer.'" Ford Motor Co.'s Mem. Regarding Source Code Protective Order, at 9, quoting Decl. of Luehrsen, at 5, ¶21, ECF No. 502-1. The email attachments unmistakably demonstrate that statement is simply untrue. Moreover, the Court finds Ford engaged in a campaign of misrepresentations that unequivocally influenced the Magistrate Judge's decision to require a level of security that, in reality, Ford did not employ itself. For instance, the Magistrate Judge rejected Plaintiffs' suggestion of a VPN between two secure rooms (one at Ford and one closer to Plaintiffs' experts) in favor of a single secure room in Dearborn, Michigan. It seems apparent now that a VPN was much less likely to be compromised than Ford's practice of emailing editable code over unsecured networks. Therefore, the Court is confident that Ford's misrepresentations caused Plaintiffs to incur unnecessary costs.
The Court does not find, however, that Plaintiffs have shown these misrepresentations would have changed the Magistrate Judge's decision that Ford must produce the source code in read-only, rather than write-access, format. Following briefing and a hearing, the Magistrate Judge analyzed this issue in depth in her June 12, 2015 Memorandum Opinion and Order. Ultimately, the Magistrate Judge determined "the benefits that Plaintiffs seek from gaining access to the source code can be realized through a read-only format." Mem. Op. and Order, at 16, ECF No. 543. The Magistrate Judge explained that "[t]he only possible limitations of a read-only format are that the source code cannot be changed and faults cannot be injected directly into the source code by editing the code." Id. at 16-17 (citation omitted). However, as pointed out at the hearing by Ford and its expert Dr. John Kelly, "changing the source code would make the resulting code unrepresentative of the code contained in Plaintiffs' vehicles. Moreover, Dr. Kelly asserted that fault injection can be performed in a way that does not implicate editing or adding lines to the source code." Id. at 17 (citations omitted).
Therefore, although the Magistrate Judge applied a proportionality analysis weighing any benefit Plaintiffs would receive against Ford's security concerns, the decision was focused on the fact a write-access format was unnecessary, and perhaps precarious, because the code could be changed, and it would no longer match what was in Ford's vehicles. In addition, the Magistrate Judge stated "it would be premature, and possibly inefficient, to grant Plaintiffs write access to the ETC source code when they have yet to even review or analyze the code in a readonly format." Id. Given these underpinnings for the Magistrate Judge's decision, any misrepresentations Ford made with respect to producing the source code in write-access format and its security concerns (beyond the fact the code could be changed) were secondary to the Magistrate Judge's decision that a read-only format was sufficient for Plaintiffs to use to perform their tests. Therefore, the Court finds Plaintiffs should not be compensated for any additional costs and fees associated with Ford producing the source code in a read-only format.
Plaintiffs further assert the source code in the emails contained segments of code they did not receive, demonstrating that Ford did not give Plaintiffs all the code the Court ordered be produced. In particular, Dr. Koopman stated it was his understanding that Ford was ordered to give Plaintiffs "all source code that directly affects the opening and closing of the throttle plate," but the source code in the secure room omitted "the code that strings the pedal to the throttle plate, such as the Level 1 Main code related to torque computation, which provides the link between Torque requests and Airflow." Decl. of Koopman, at 5-6, ¶18.
Dr. Koopman said "it is impossible to fully analyze the operation of the throttle plate" without those sections of the code. Id.
In reviewing the transcripts and orders by the Magistrate Judge, the Court finds Dr. Koopman was mistaken in what Ford was directed to produce. Specifically, on April 3, 2015, the Magistrate Judge entered an Order directing Ford to
Mem. Op. and Order, at 9-10 (April 3, 2015), ECF No. 421. Similarly, during a telephonic discovery conference held on November 13, 2015, the Magistrate Judge stated that Plaintiffs "should be allowed to look at the source code of whatever is in the PCM that directly affects the opening and closing of the throttle plate." Tr. of Tel. Disc. Conf., at 10 (Nov. 13, 2015), ECF No. 677.
After making these decisions, the parties continued to vigorously contest precisely what source code should be produced. As the parties' arguments were extremely complex and involved highly technical aspects of computer engineering, the Magistrate Judge, with the consent of the parties, appointed William H. Sanders, Ph.D, M.S.E., B.S.E., in April of 2016 as an independent technical advisor to assist with source code discovery issues. Order Appointing Tech. Advisor for the Court, ECF No. 740. With the consent of the parties, this Court also has utilized Dr. Sanders as a technical advisor with respect to the current motion.
On August 12, 2016, Dr. Sanders wrote a report discussing a variety of issues related to the source code and other matters. Report Concerning Code, Features, and Drivers Produced by Ford and Request by Pls.' Concerning Additional Software Tools, (Aug. 12, 2016), ECF No. 793. After reviewing the report, the Magistrate Judge entered an Order on September 1, 2016, stating, in part:
Mem. Op. and Order Regarding Source Code Disc., at 6 (Sept. 1, 2016), ECF No. 800. In addition, Magistrate Judge Eifert denied Plaintiffs' request for global variables because they "are not considered primary to ETC functionality" and are unnecessary to test the fault tolerance of the ETC system. Id. at 4.
Ford insists that it fully complied with these decisions and provided Plaintiffs with all the source code it was required to produce. Ford insists the production sought by Plaintiffs included features that do not directly open or close the throttle. Rather, they calculate the global variables that Magistrate Judge Eifert ruled it did not have to produce.
Upon review, the Court agrees with Ford. It is clear from the Magistrate Judge's Orders and hearings with the parties that Ford was not ordered to produce the source code related to global variables. Thus, Dr. Koopman's understanding that Ford was to produce all the source code is mistaken. The Magistrate Judge determined the amount of code produced by Ford was sufficient for Plaintiffs to test for faults consistent with the theory of their case that the ETC system is not fault tolerant. In fact, after given the opportunity to review and test the source code in the secure room, Plaintiffs have never alleged any source code Ford was ordered to produce was missing until they filed the current motion for sanctions.
Plaintiffs contend, however, that Ford wildly exaggerated the amount of time it would take to produce all the source code. Thus, the Magistrate Judge, in part, based her decision to deny production of the global variables on a misrepresentation by Ford. Specifically, Ford stated it would take 150 hours per model and model year to produce all the source code. See Decl. of Thomas C. Erickson, at 2, ¶11 (Nov. 6, 2015), ECF No. 665-1. To the contrary, Plaintiffs contend they found an email in which an employee received a request for source code at 4:44 p.m., and it was produced by the close of business the following day.
Upon consideration, Plaintiffs have not convinced the Court that this single email is representative of the amount of time it would take to produce the source code provided in the secure room. While it seems reasonable to assume that, once Ford supplied the code and the other tools necessary for the first vehicle, the process should go faster for the subsequent vehicles, it is likely that it still would take a significant amount of time to produce all the source code Plaintiffs requested. Moreover, although the Magistrate Judge considered Ford's proportionality argument, the crux of her decision was based upon the fact the global variables are unnecessary for Plaintiffs to test their theory of their case. See Mem. Op. and Order Regarding Source Code Disc., at 6 (stating "[t]hese parameters were selected by the court, because Plaintiffs' theory has always been that Ford's ETC system is not fault tolerant, regardless of the nature of the faults. Therefore, identifying specific faults (such as interaction faults) should not be the focus of discovery. Discovery in this case should center on how the ETC functionality—including its watchdogs and fail-safes—works when simultaneously confronted with multiple faults."). Accordingly, even if Ford overestimated the amount of time it would take to produce the code, the Court finds it would not have fundamentally changed the Magistrate Judge's decision.
In sum, the Court finds Plaintiffs have failed to establish that Ford violated any Orders with respect to the extent of the source code it was directed to produce, but it did make statements to the Magistrate Judge that it knew, or should have known, were false with regard to the level of security that Ford, in practice, used to protect the code. These misrepresentations indisputably resulted in Plaintiffs unnecessarily spending additional time and money to conduct discovery to which they were entitled. Despite Ford's insistence it cannot be sanctioned under Rule 37 because Plaintiffs cannot demonstrate, as a threshold matter, that Ford violated any Order of the Court, it is well established that this Court can exercise its inherent power "to fill in the interstices" not covered by the Rules or a statute. Chambers, 501 U.S. at 46. Given the material misrepresentations Ford made in this case, as set forth above, the Court finds that an award of attorneys' fees, expert fees, and costs are warranted to compensate Plaintiffs for any reasonable expense they can demonstrate was attributable to Ford's misrepresentations related to the level of security it used to protect its source code. See Goodyear Tire & Rubber Co., 137 S. Ct. at 1189 (stating "[a] sanctioning court must determine which fees were incurred because of, and solely because of, the misconduct at issue"). At this point, the Court does not know what those fees, expenses, and costs are, and it finds that the Magistrate Judge is the best person to make that decision. Therefore, the Court
Accordingly, for the foregoing reasons, the Court finds Ford was not forthright with respect to the level of security it provided its source code, which resulted in the Magistrate Judge unnecessarily requiring Plaintiffs to limit their review of the code to a secure facility in Michigan. However, the Court also finds that Ford provided all the source code it was ordered to produce by the Magistrate Judge and, to the extent Ford made other misrepresentations to the Court, those misrepresentations did not materially affect the Magistrate Judge's decisions. Therefore, the Court
The Court
Mut. Fed. Sav. & Loan Ass'n v. Richards & Assocs., Inc., 872 F.2d 88, 92 (4th Cir. 1989) (citing Wilson v. Volkswagen of Am., Inc., 561 F.2d 494, 503-06 (4th Cir. 1977)). The Fourth Circuit explained that these considerations "will insure that only the most flagrant case, where the party's noncompliance represents bad faith and callous disregard for the authority of the district court and the Rules, will result in the extreme sanction of dismissal or judgment by default." Id. (citation omitted). The Court agrees with Ford that the facts of this case are not so egregious as to warrant such a drastic sanction.
On the other hand, Plaintiffs state they did not discover the source code attached to the emails until April 15, 2017. Although Plaintiffs had a significant number of other activities and deadlines to meet in this case, they were able to file the current, highly technical motion in two months. In fact, Plaintiffs point out that Ford got more time to file its Response than Plaintiffs took to file the motion.
Although Rule 37(b) does not contain express time limits, a few courts have said that a motion for sanctions may be untimely if made after unreasonable delay. See Brandt v. Vulcan, Inc., 30 F.3d 752, 756 (7th Cir. 1994) (applying an "unreasonable delay" standard to Rule 37(b) where motion for sanctions was made at jury instruction conference); Mercy v. County of Suffolk, New York, 748 F.2d 52, 55-56 (2d Cir. 1984) (stating "a motion for Rule 37 sanctions should be promptly made, thereby allowing the judge to rule on the matter when it is still fresh in his mind. . . . Indeed, the motion should normally be deemed waived if it is not made prior to trial"); Shamis v. Ambassador Factors Corp., 34 F.Supp.2d 879, 886 (S.D.N.Y. 1999) ("While Rule 37 does not establish any time limits within which a motion for sanctions must be filed, unreasonable delay may render such motions untimely."). "The timeliness of a motion for sanctions depends on such factors as when the movant learned of the discovery violation, how long he waited before bringing it to the court's attention, and whether discovery has been completed." Long v. Howard Univ., 561 F.Supp.2d 85, 91 (D. D.C. 2008) (citations omitted). Applying these factors to the facts of this case, the Court agrees with Plaintiffs that their motion is not untimely. In addition, the Court agrees with Plaintiffs that it was unnecessary for them to meet and confer with Ford with respect to this specific motion.