SPECIAL MASTER DANIEL B. GARRIE E-DISCOVERY SUMMARY AND ORDER
DANIEL B. GARRIE, District Judge.
BACKGROUND AND PRIOR PROCEEDINGS
The Special Master was appointed on March 3, 2014. (Dkt. No. 149.) On March 10, 2014, the parties, counsel for all parties, and ESI consultants for all parties, met with Special Master Daniel Garrie and United States Magistrate Judge Peggy Leen in chambers. (Dkt. No. 151.) On March 18, 2014, Special Master Garrie memorialized his directives to the parties in a written order. (Dkt. No. 154.)
On March 18, 2014 Order Special Master Garrie set a hearing schedule, with hearings on April 4, 2014 and April 7, 2014. Present on behalf of Plaintiffs were Counsel Jon Tostrud, Marc Godino, and David O'Mara and ESI consultants Douglas Forrest and Bruce Pixley. Present on behalf of Defendant were Counsel Margaret Foley and Cayla Witty, ESI vendor Joe Edmondson, and Dean Schaibley, Network Security Administrator from the UMC IT Department. In addition, the following UMC IT individuals attended the April 7, 2014 hearing: Jason Clark, Sr. Systems Administrator, Marilyn Susan Kisner, IT Customer Support Manager, and Shane Lattin, Network Engineer. See Exhibit A (April 4, 2014 rough hearing transcript); Exhibit B (April 7, 2014, hearing transcript.)
At the April 7, 2014 hearing, Special Master Garrie ordered an additional telephone hearing on April 10, 2014. See Exhibit B at 234:01-236:21 Present at the April 10, 2014 telephonic hearing were Counsel and their respective ESI Consultants and ESI vendor, and no individuals from UMC's IT Department. See Exhibit C (April 10, 2014, rough hearing transcript)
The Special Master conducted full day hearings on April 4, 2014 and April 7, 2014 with counsel, the parties' representatives, and consultants regarding UMC's ESI collection and production issues, as well as UMC's efforts to preserve discoverable materials pursuant to the Plaintiffs' litigation hold/preservation letters, and UMC's search of the ESI collected.
These hearings identified multiple potential failures by UMC with respect to the collection, preservation, and search of the ESI by UMC.
ISSUES WITH UMC COLLECTION EFFORTS
A. UMC ESI Collection was Incomplete as of April 10, 2014.
Among other things.
1. UMC did not collect the laptops of Doug Spring, Director of Personnel Operations, and John Mumford, Sr. Human Resources Analyst, both potential sources of relevant ESI. See Exhibit D, UMC Custodian Interviews (Both John Mumford and Doug Spring stated that they had personal laptops that they used to conduct UMC business, including SEIU labor negotiations in 2009.)
2. UMC did not include Claudette Myers who is the Executive Assistant to John Espinoza, Chief Human Resources Officer at UMC, in the initial custodian group even though Mr. Espinoza stated in his custodian interview that Ms. Myers maintained electronic filing, regularly accessed both Mr. Espinoza's calendar and e-mail, archived Mr. Espinoza's e-mail and calendars, and sent email and documents on Mr. Espinoza's behalf. See Exhibit D, 15 to 17.
3. UMC never collected ESI from any UMC network file shares. See Exhibit B (hearing transcript from 04/07/14) at 116:2-10 (Mr. Schaibley states that UMC did not collect any of the network file shares.) This means that UMC never collected any information from the UMC payroll network file share. See Exhibit D (custodian interviews) at 8, 13, 21, and 22 (Custodian interviews taken of John Mumford, Brian Brannman, John Mumford, and Doug Spring all indicate that they stored documents in UMC network file shares.)
4. UMC did not collect ESI from the UMC Blackberry server for any of the twenty seven custodians. See Exhibit B (hearing transcript from 04/07/14) at 63 to 67 (Mr. Clark states that he was never instructed to preserve or collect communications or data from the UMC Blackberry server environment.)
5. UMC failed to review the collection scripts, and, as a result, failed to identify errors that occurred in the collection. See Exhibit A (hearing transcript from 04/04/14) at 43:1-5 (Counsel Foley stated that she was never informed of collection errors by UMC or UMC ESI vendor.)
6. UMC's IT staff did not verify that they had provided to UMC's ESI vendor all the data they had collected. See Exhibit B (hearing transcript from 04/07/14) at 138:24-25; 139:1-15 (Mr. Edmonson indicates not all of UMC ESI collection was searched by him in response to the multiple production requests and attempts.) The Special Master notes that, because not all ESI collected by UMC was searched for production, the entire production discussion before the Magistrate Judge Leen was based on a false premise.
B. UMC Failed to Perform Custodian Interviews.
UMC failed to perform custodian interviews and this contributed to many of the above collection issues. See Exhibit A (hearing transcript from 04/04/14) at 50:3-25; 51:1-5 (UMC states that no custodian interviews were performed by it on or before the Special Master proceedings.)
C. UMC and UMC ESI Vendor Did Not Create Any Chain of Custody Paper Work for any of the ESI that was Collected by the UMC IT Department.
On March 18, 2014, Special Master Garrie ordered UMC to create Chain-of-Custody paper work before the hearing on April 4, 2014. While UMC did submit a chain-of-custody, it was inadequate for its purpose. See Exhibit A (hearing transcript from 04/04/14) from 46 to 48. Specifically, the chain-of-custody failed to record several things, including (i) what sources UMC collected data from for each of the twenty seven (27) custodians, (ii) how they identified the ESI, (iii) how they collected the ESI, (iv) how the ESI was preserved, (v) what criteria were used to identify the ESI that was collected.
ISSUES WITH UMC PRESERVATION EFFORTS
UMC was informed by Plaintiffs and the Court multiple times of its obligation to preserve information, including on: August 6, 2012, November 6, 2014, March 10, 2014, March 18, 2014 and April 4, 2014. See e.g., Exhibit E (preservation letter sent to UMC); Exhibit F (internal UMC email informing patient service leaders to preserve documents); Exhibit G (deposition testimony of Mr. Espinoza where he states he is aware of his duty to preserve documents). Despite this it appears that UMC did not take the necessary steps required to ensure preservation of relevant ESI by all custodians. See Exhibit B (hearing transcript from 04/07/14) 84:1-15 (Ms. Kisner states she did not receive a preservation notice and yet Ms. Kisner is one of the UMC IT employees responsible for wiping blackberry devices.)
It appears that UMC has not preserved:
1. Data on the UMC Intranet. See Exhibit B (hearing transcript from 04/07/14) at 45:10-18; 46:12-20 (UMC states that did not collect or preserve ESI that existed on its intranet.)
2. UMC network file shares. See Exhibit B (hearing transcript from 04/07/14) at 45:10-18; 46:12-20 (UMC states that did not collect or preserve ESI that existed on the network file shares.)
3. UMC e-mail and messages stored on the blackberry server. See Exhibit B (hearing transcript from 04/07/14) at 36:3-25; 63:21-25; 64:1-10; 65:1-23; 83:6-18 (Multiple UMC IT stakeholders state that they have not yet preserved the blackberry server data as of April 7, 2014, and only gave the IT individual who could effectuate preservation a notice two weeks ago.)
4. UMC failed to preserve the computers of the twenty seven (27) UMC custodians. See Exhibit B (hearing transcript from 04/07/2014) at 112:1-18, 127-129; Exhibit H at 9 (Declaration from Dan Small that includes as an exhibit an e-mail sent to all UMC employees that states many UMC employees store documents on their local computers.)
In addition, UMC is still unable to answer questions as to what data preservation policies have been implemented on their blackberry server and intranet. See Exhibit B (hearing transcript from 04/07/2014) at 98-102 (Several UMC IT employees state they do not know the policies implemented on UMC Blackberry server or intranet.)
ISSUES WITH UMC FIRST, SECOND, AND THIRD PRODUCTION
The UMC productions suffered from certain technical issues that prevented appropriate search and collection. See Exhibit I (UMC ESI vendor findings regarding production) at 3 (states that "[d]espite the manufacturers claims of compatibility the results of these tests show that it is likely that most errors were caused by P2 Commander not processing the source files accurately.") It appears that the software tool chosen by UMC, while used in the industry, did not work. Special Master Garrie was able to resolve these technical issues.
In addition, UMC failed to identify several encrypted files in production, which were not searched. See Exhibit B (transcript of hearing on 04/07/2014) at 26-29 (UMC ESI vendor states there were various encrypted files which included two DMG files for Macintosh, which UMC is unable to explain since none of the custodians supposedly had or used Macintosh devices.)
POSSIBLE SPOLIATION ISSUES
UMC has yet to accomplish a complete collection of all responsive ESI. This fact, together with the issues identified above, suggests potential spoliation of relevant evidence. Spoliation will be the subject of further attention by Special Master Garrie, and will be one of the topics addressed in his final order.
IT IS HEREBY FURTHER ORDERED THAT Plaintiffs and UMC will have done or do the following:
Plaintiffs.
1. Plaintiffs will have provided written questions regarding custodian interviews in advance of the phone hearing on April 10, 2014.
2. Plaintiffs are to be prepared to discuss, at the telephonic hearing on April 15, 2014, the file types to be searched by UMC.
3. Plaintiffs are to provide the Special Master with a letter or affidavit explaining why they believe Lonnie Richardson, Ernie McKinley, and Lawrence Bernard should be added to list of custodians.
4. Plaintiffs will have provided, by April 11, 2014, to Special Master Garrie results of UMC SAP data analysis from 10 opt-in packets.
5. Plaintiffs will file with the Court, by April 17, 2014, the amended ESI Protocol.
Defendants.
1. UMC to start producing data to Plaintiffs' on or before April 22, 2014 at 10:30am PST, starting with custodian John Espinoza. UMC production is to follow the amended ESI Protocol.
2. UMC ESI expert is to provide Special Master Garrie and Plaintiffs with a spreadsheet on or before April 14, 2014. The spreadsheet shall set-forth a count of the total number of emails that were lost in the recovery process.
3. UMC is to provide the Special Master with the following on or before April 16, 2014, unless noted otherwise:
• UMC archiving policy for the data stored on the UMC intranet.
• Name and contact details of the individual(s) in UMC IT department who implement the record/data retention schedules for emails, Blackberry communications, and SAP/Kronos systems.
• Specific pages of UMC retention and deletion policies/guidelines for the following systems: network file-shares, e-mail, blackberry devices blackberry server, SAP server, Kronos, and any other internal systems used by the custodians.
• Complete list of all UMC custodians that have smartphones that are either UMC issued or personal that are used to conduct UMC business.
• Spreadsheet that details the network file share mappings of the initial six custodians by April 15, 2014, and the remaining custodians by April 21, 2014.
4. UMC is to submit by April 15, 2014, the following documents, unless noted otherwise:
• Explanation of UMC Blackberry server configuration during the time period in question.
• Clarification of how UMC Blackberry environment operates with UMC Exchange environment during the time period in question.
• Explanation of McAfee configuration during the time period in question.
• Enhanced data map that includes the data repositories at UMC clinics.
• USB connectivity logs for the initial six custodians by April 18, 2014.
• Spreadsheet that lists which of the twenty seven (27) UMC custodians are using Windows XP on their desktop today. In addition, the spreadsheet should identify which of these UMC custodians' computers, laptops, or other devices have been updated since 2011. The spreadsheet also should identify any computers or devices these custodians access more than 5 times a month.
• Affidavit or declaration from UMC employee Ms. Kisner that identifies all wiping that occurred with respect to the mobile devices for each of the twenty seven (27) custodians, including personal smartphones, on or before April 18, 2014. For the sake of clarity, Ms. Kisner should identify if she or anyone on her team has ever configured or been asked to configure a smartphone to access UMC e-mail system for any of the twenty seven (27) custodians, including personal smartphones.
• UMC is to provide on or before April 18, 2014 a spreadsheet that lists all requests made to UMC internal IT support/help desk system to assist with mobile devices or to create backups (e.g., burning CD/DVD) for each of the twenty seven custodians for the time period in question.
• Screen shot or affidavit identifying if UMC OWM configuration sets a flag at the server level not to cache the body of the messages to the web client.
• Spreadsheet of all network file share folders created by UMC employees or consultants in connection with the Department of Labor investigation, and it also should identify which custodians had access to these folders.
• Detailed log files that show when UMC employees accessed UMC systems via the web or VPN during the relevant time period.
• A letter stating the initial date of the Department of Labor investigation.
• An affidavit of declaration by April 17, 2014 that updates the UMC ESI vendor's Scan/Repair findings to reflect the correct number of OST and PST files for the initial custodians, and it is to include an explanation as to why this error occurred.
• UMC is to work with the Special Master to determine if the initial six UMC custodians used personal mobile devices to conduct UMC business, and be prepared to discuss at the April 15, 2014 telephonic hearing.1
5. UMC is to provide, on or before April 16, 2014, a letter to Special Master Garrie and Plaintiffs a letter affirming that, on a going forward basis, the following ESI repositories are being preserved using industry best practices:
• All data on e-mail lists,
• Data stored on Blackberry server for the twenty seven (27) custodians
• All network file files shares that contain responsive data, including: Department of Labor investigation folder;
• Doug Spring UMC issued laptop
• James Mumford personal laptop that was used for UMC business
• All UMC computers, laptops, tablets, or other devices used by twenty seven custodians because of UMC email.
• All network files shares, including those identified in the custodian interviews provided to Special Master Garrie.
• UMC IIS servers, lotus notes/domino server, and any other server used by UMC intranet.
6. UMC is to amend the chain of custody forms and submit them to Special Master Garrie and the Plaintiffs by April 14, 2014. The amended chain of custody forms will include the following: details as to what information was collected by UMC for each custodian; identify what ESI sources UMC collected from for each custodian; UMC is to update the chain of custody forms to include details for each mobile device it collected; and MD5 hash values for each evidence container that was collected.
7. UMC is to provide a declaration or affidavit, on or before April 18, 2014, to Special Master Garrie from the original e-discovery provider that addresses, the following: (i) how they failed to properly collect the ESI that UMC initially collected; (ii) why they were unable to collect in a forensically sound manner the ESI gathered by UMC either in April and August. UMC Counsel is to submit a two page letter to Special Master Garrie on or before April 21, 2014, that explains how UMC Security individual, UMC Counsel, UMC current forensic expert, and UMC former forensic expert did not realize the ESI collection being searched was not the entire ESI repository collected by UMC. In addition, UMC ESI expert is to submit an affidavit on or before April 25, 2014, which identifies each and every file that was collected by UMC and not included in UMC's first, second, or third ESI production.
8. UMC is to identify the personal email addresses for the six initial custodians, and run a search for these email addresses and review the results to determine if these custodians were using their personal email address to conduct UMC business. UMC is to notify the Special Master, on or before April 21, 2014, of the results of the search.
9. UMC counsel is to review Plaintiffs' prior request for documents, search the updated UMC ESI collection, and produce any responsive ESI on a rolling basis starting no later than April 24, 2014.
HEARING DATES
The following dates for hearings are as follows: April 15, April 22, and May 7. All hearings will be held at the Las Vegas court house or telephonically with a court reporter present. The Special Master will coordinate with the court to confirm space is available. The Special Master Orders the parties to be PREPARED and their ESI and UMC IT EXPERTS TO BE AVAILABLE either by phone or in-person.
At the hearing on April 22, 2014, the Special Master has ordered Lawrence (Larry) Barnard, Chief Executive Officer, Ernie McKinley, Chief Information Officer, and John Espinoza, Chief Human Resources Officer to appear at the hearing, their presence being necessary to provide a complete understanding of UMC's discovery related actions and inactions to date.
The Special Master Orders the parties to provide all materials to be discussed at the hearing at least 24 hours prior to the hearing.
TIME ESTIMATE
The Special Master at this time is unable to provide a precise estimate as to the amount of time required to perform the work set-forth above.
SO ORDERED.
