ORDER

WARDLAW and GOULD, Circuit Judges:

On September 5, 2017, the Acting Secretary of the Department of Homeland Security ("DHS"), Elaine Duke, announced the end of DHS's Deferred Action for Childhood Arrivals policy ("DACA"), effective March 5, 2018. Begun in 2012, DACA provided deferred action for certain individuals without lawful immigration status who had entered the United States as children. Several sets of plaintiffs sued to enjoin the rescission of DACA under the Administrative Procedure Act ("APA") and under various constitutional theories not relevant here.

The merits of those claims are not before us today. The only issue is a procedural one, raised by the government's petition for a writ of mandamus. The government asks us to permanently stay the district court's order of October 17, 2017, which required it to complete the administrative record.1 See Order re Motion to Complete Administrative Record, Regents of the Univ. of Cal. v. U.S. Dep't of Homeland Sec., No. C 17-05211 WHA, 2017 WL 4642324 (October 17, 2017) ("Order"). We have jurisdiction pursuant to the All Writs Act, 28 U.S.C. § 1651. Because the district court did not clearly err by ordering the completion of the administrative record, we hold that the government has not met the high bar required for mandamus relief.

One note at the outset: We are not unmindful of the separation-of-powers concerns raised by the government. However, the narrow question presented here simply does not implicate those concerns. We consider only whether DHS failed to comply with its obligation under the APA to provide a complete administrative record to the court — or, more precisely, whether the district court clearly erred in so holding. See Citizens to Preserve Overton Park, Inc. v. Volpe, 401 U.S. 402, 420, 91 S.Ct. 814, 28 L.Ed.2d 136 (1971) ("[R]eview is to be based on the full administrative record that was before the Secretary at the time he made his decision."). This obligation is imposed to ensure that agency action does not become effectively unreviewable, for "[i]f the record is not complete, then the requirement that the agency decision be supported by `the record' becomes almost meaningless." Portland Audubon Soc'y v. Endangered Species Comm., 984 F.2d 1534, 1548 (9th Cir. 1993). Assuring that DHS complies with this requirement — imposed by the APA on all agencies and embodied in decades of precedent — is undoubtedly a proper judicial function.

1. "The writ of mandamus is a drastic and extraordinary remedy reserved only for really extraordinary cases." In re Van Dusen, 654 F.3d 838, 840 (9th Cir. 2011) (quoting Ex parte Fahey, 332 U.S. 258, 259-60, 67 S.Ct. 1558, 91 S.Ct. 2041 (1947) (internal quotation marks omitted)). Indeed, "only exceptional circumstances amounting to a judicial usurpation of power or a clear abuse of discretion will justify the invocation of this extraordinary remedy." Cheney v. U.S. Dist. Court, 542 U.S. 367, 380, 124 S.Ct. 2576, 159 L.Ed.2d 459 (2004) (internal quotation marks and citations omitted). Ultimately, the issuance of the writ is "in large measure ... a matter of the court's discretion." Johnson v. Consumerinfo.com, Inc., 745 F.3d 1019, 1023 (9th Cir. 2014) (quoting United States v. Sherman, 581 F.2d 1358, 1361 (9th Cir. 1978)).

Our discretion is guided by the five factors laid out in Bauman v. U.S. District Court, 557 F.2d 650 (9th Cir. 1977). However, we need not consider four of those five factors here, because "the absence of factor three — clear error as a matter of law — will always defeat a petition for mandamus." In re Bundy, 840 F.3d 1034, 1041 (9th Cir. 2016) (quoting In re United States, 791 F.3d 945, 955 (9th Cir. 2015)). This factor — whether "[t]he district court's order is clearly erroneous as a matter of law," Bauman, 557 F.2d at 654-55 — "is significantly deferential and is not met unless the reviewing court is left with a definite and firm conviction that a mistake has been committed." In re Bundy, 840 F.3d at 1041 (quoting In re United States, 791 F.3d at 955).

2. The district court's order is not clearly erroneous as a matter of law. APA § 706 provides that arbitrary and capricious review shall be based upon "the whole record or those parts of it cited by a party." 5 U.S.C. § 706. The whole record "includes everything that was before the agency pertaining to the merits of its decision." Portland Audubon, 984 F.2d at 1548; see also, e.g., James Madison Ltd. by Hecht v. Ludwig, 82 F.3d 1085, 1095 (D.C. Cir. 1996) ("The administrative record includes all materials compiled by the agency that were before the agency at the time the decision was made.") (internal quotation marks and citations omitted). More specifically, we have explained that the whole administrative record "consists of all documents and materials directly or indirectly considered by agency decision-makers and includes evidence contrary to the agency's position." Thompson v. U.S. Dep't of Labor, 885 F.2d 551, 555 (9th Cir. 1989) (internal quotation marks omitted); see also Bar MK Ranches v. Yuetter, 994 F.2d 735, 739 (10th Cir. 1993) (same). The record is thus not necessarily limited to "those documents that the agency has compiled and submitted as `the' administrative record." Thompson, 885 F.2d at 555 (internal quotation marks omitted).

At the initial case management conference before the district court, the government agreed to produce the complete administrative record on October 6, 2017. On that date, the government submitted as "the" administrative record fourteen documents comprising a mere 256 pages, all of which are publicly available on the internet. Indeed, all of the documents in the government's proffered record had previously been included in filings in the district court in this case, and 192 of its 256 pages consist of the Supreme Court, Fifth Circuit, and district court opinions in the Texas v. United States litigation.2

Faced with this sparse record, and on the plaintiffs' motion (opposed by the government), the district court ordered the government to complete the record to include, among other things, all DACA-related materials considered by subordinates or other government personnel who then provided written or verbal input directly to Acting Secretary Duke. The district court excluded from the record documents that it determined in camera are protected by privilege. Order at *8.

3. The administrative record submitted by the government is entitled to a presumption of completeness which may be rebutted by clear evidence to the contrary. Bar MK Ranches, 994 F.2d at 740; see also Thompson, 885 F.2d at 555 (noting that the administrative record "is not necessarily those documents that the agency has compiled and submitted as `the' administrative record."). The district court correctly stated this legal framework and concluded that the presumption of completeness had been rebutted here. Order at *5. This conclusion was not clear legal error: Put bluntly, the notion that the head of a United States agency would decide to terminate a program giving legal protections to roughly 800,000 people3 based solely on 256 pages of publicly available documents is not credible, as the district court concluded.4

The district court identified several specific categories of materials that were likely considered by the Acting Secretary or those advising her, but which were not included in the government's proffered record. For example, the record contains no materials from the Department of Justice or the White House — other than a one-page letter from Attorney General Jefferson B. Sessions — despite evidence that both bodies were involved in the decision to end DACA, including the President's own press release taking credit for the decision.5 Nor does the proffered record include any documents from Acting Secretary Duke's subordinates; we agree with the district court that "it strains credulity" to suggest that the Acting Secretary decided to terminate DACA "without consulting one advisor or subordinate within DHS." Order at *4. And the proffered record contains no materials addressing the change of position between February 2017 — when then-Secretary John Kelly affirmatively decided not to end DACA — and Acting Secretary Duke's September 2017 decision to do the exact opposite, despite the principle that reasoned agency decision-making "ordinarily demand[s] that [the agency] display awareness that it is changing position" and "show that there are good reasons for the new policy." F.C.C. v. Fox Television Stations, Inc., 556 U.S. 502, 515, 129 S.Ct. 1800, 173 L.Ed.2d 738 (2009).

At oral argument, the government took the position that because the Acting Secretary's stated justification for her decision was litigation risk, materials unrelated to litigation risk need not be included in the administrative record. Simply put, this is not what the law dictates. The administrative record consists of all materials "considered by agency decision-makers," Thompson, 885 F.2d at 555 (emphasis added), not just those which support or form the basis for the agency's ultimate decision. See also, e.g., Amfac Resorts, LLC v. U.S. Dep't of Interior, 143 F.Supp.2d 7, 12 (D.D.C. 2001) ("[A] complete administrative record should include all materials that `might have influenced the agency's decision,' and not merely those on which the agency relied in its final decision.") (quoting Bethlehem Steel v. EPA, 638 F.2d 994, 1000 (7th Cir. 1980)). And even if the record were properly limited to materials relating to litigation risk, the district court did not clearly err in concluding that it is implausible that the Acting Secretary would make a litigation-risk decision "without having generated any materials analyzing the lawsuit or other factors militating in favor of and against the switch in policy." Order at *4.

It was therefore not clear error for the district court to conclude that the presumption of regularity that attaches to the government's proffered record is rebutted, and that ordering completion of the record was necessary and appropriate.

4. Nor did the district court clearly err in identifying the materials that should have been included within the scope of the complete administrative record. The government challenges the decision to include materials considered by subordinates who then briefed the Acting Secretary, but this decision was not clear legal error. We have held that the record properly includes "all documents and materials directly or indirectly considered by agency decision-makers," Thompson, 885 F.2d at 555, but have not yet clarified the exact scope of "indirectly considered." District courts in this and other circuits, however, have interpreted that phrase to include materials relied on by subordinates who directly advised the ultimate decision-maker. See, e.g., Nat. Res. Def. Council v. Gutierrez, No. C 01-0421 JL, 2008 WL 11358008, at *6 (N.D. Cal. Jan. 14, 2008) ("To the extent [the government argues] that only those documents that reached [the agency's] most senior administrators were in fact `considered,' courts have rejected that view as contrary to the Ninth and other Circuits' pronouncements...."); GeorgiaCarry.org, Inc. v. U.S. Army Corps of Eng'rs, 212 F.Supp.3d 1348, 1352 (N.D. Ga. 2016) ("Documents and materials indirectly considered by agency decision-makers are those that may not have literally passed before the eyes of the decision-makers, but were so heavily relied on in the recommendation that the decisionmaker constructively considered them."); Amfac Resorts, 143 F.Supp.2d at 12 ("[I]f the agency decisionmaker based his decision on the work and recommendations of subordinates, those materials should be included as well.").6

Moreover, as noted in the district court's October 17 order, a Department of Justice guidance document directs agencies compiling the administrative record to "[i]nclude all documents and materials prepared, reviewed, or received by agency personnel and used by or available to the decision-maker, even though the final decision-maker did not actually review or know about the documents and materials." U.S. Dep't of Justice, Env't and Nat. Res. Div., Guidance to Federal Agencies on Compiling the Administrative Record 3 (Jan. 1999) (emphasis added). It further provides that the administrative record should include "communications the agency received from other agencies ... documents and materials that support or oppose the challenged agency decision ... minutes of meetings or transcripts thereof... [and] memorializations of telephone conversations and meetings, such as a memorandum or handwritten notes." Id. at 3-4. The district court's October 17 order complies with this Department of Justice guidance; the government's proffered record does not.

We recognize that such guidance is not binding; we nevertheless find it persuasive as a statement by the Department of Justice as to what should be included in a complete administrative record. We also note that the guidance document DHS failed to comply with here was inexplicably rescinded the very same day that the government filed this petition for a writ of mandamus.

Given that the district court's interpretation of Thompson is consistent with the rulings of other district courts, comports with the Department of Justice's guidance on administrative records, and is not foreclosed by Ninth Circuit authority, we cannot say that the district court's interpretation was clearly erroneous as a matter of law. See In re Swift Transp. Co., 830 F.3d 913, 916-17 (9th Cir. 2016) ("It is well established that `[t]he absence of controlling precedent weighs strongly against a finding of clear error [for mandamus purposes].'") (quoting In re Van Dusen, 654 F.3d at 845).7

5. The district court's order that the government complete the record with documents considered by former DHS Secretary John Kelly in the course of deciding not to terminate DACA in February 2017 also withstands mandamus scrutiny. This is not because of some freestanding requirement that all the materials underlying a previous decision on a similar subject are always part of the administrative record; rather, it simply recognizes that both decisions were part of an ongoing decision-making process regarding deferred action: In February 2017, Secretary Kelly ended other prioritization programs, but left DACA and DAPA in place; in June 2017, Secretary Kelly ended DAPA but left DACA intact; finally, in September 2017, Acting Secretary Duke ended DACA. The materials considered by Secretary Kelly in the course of deciding against ending DACA in February 2017 did not cease to be "before the agency" for purposes of the administrative record during that seven-month evolution in policy. Thompson, 885 F.2d at 555-56. The district court's decision to order their inclusion in the record was therefore not clear legal error.

6. Finally, the government makes two categorical arguments with respect to privilege.8 First, it contends that Cheney v. U.S. District Court, 542 U.S. 367, 124 S.Ct. 2576, 159 L.Ed.2d 459 (2004), bars the completion of the administrative record with any White House materials, because requiring White House officials to search for and assert privilege as to individual documents would be an unwarranted intrusion into executive decision-making. Cheney, of course, did not involve an administrative agency's obligation under the APA to provide the court with the record underlying its decision-making. It instead involved civil discovery requests that the Supreme Court described variously as "overbroad" and as "ask[ing] for everything under the sky." Id. at 383, 387, 124 S.Ct. 2576. We do not read Cheney as imposing a categorical bar against requiring DHS to either include White House documents in a properly-defined administrative record or assert privilege individually as to those documents.

Moreover, the reasoning of Cheney appears to be based substantially on the fact that the Vice President himself was the subject of discovery. See id. at 381, 124 S.Ct. 2576 ("Here, however, the Vice President and his comembers on the NEPDG are the subjects of the discovery orders."), 382 ("These separation-of-powers considerations should inform a court of appeals' evaluation of a mandamus petition involving the President or the Vice President."). Here, although the government is of course correct that the President is named as a defendant in some of the underlying lawsuits, there is no indication that either his documents or those of the Vice President would fall within the completed administrative record as ordered by the district court. Cheney therefore does not render the district court's order clearly erroneous.

Second, the government argues that it was clear legal error to require a privilege log and to evaluate documents allegedly protected by the deliberative process privilege on an individual basis, since "deliberative" materials are not properly within the administrative record at all. As noted above, the district court reviewed in camera each of the documents as to which the government asserted the deliberative process privilege, and ordered the inclusion of only those documents that met the balancing standard laid out in FTC v. Warner Commc'ns, Inc., 742 F.2d 1156, 1161 (9th Cir. 1984). The court stated that it would similarly review in camera any additional documents as to which the government claims privilege in the future. Order at *8.

As the government acknowledges, we have not previously addressed whether assertedly deliberative documents must be logged and examined or whether the government may exclude them from the administrative record altogether. However, many district courts within this circuit have required a privilege log and in camera analysis of assertedly deliberative materials in APA cases. See, e.g., Ctr. for Food Safety v. Vilsack, No. 15-cv-01590, 2017 WL 1709318, at *5 (N.D. Cal. May 3, 2017); Inst. for Fisheries Res. v. Burwell, No. 16-cv-01574 VC, 2017 WL 89003, at *1 (N.D. Cal. Jan. 10, 2017); California ex rel. Lockyer v. U.S. Dep't of Agric., No. C05-03508 EDL, 2006 WL 708914, at *4 (N.D. Cal. March 16, 2008). Again, "the absence of controlling precedent" and the practice of the district courts "weigh[] strongly against a finding of clear error" for purposes of mandamus. In re Swift Transp. Co., 830 F.3d at 916-17 (citation omitted).

We further note that the "deliberative" materials at issue in the main case cited by the government, San Luis Obispo Mothers for Peace v. U.S. Nuclear Regulatory Comm'n, 789 F.2d 26 (D.C. Cir. 1986) (en banc), were transcripts of literal deliberations among the members of a multi-member agency board. See id. at 44. Where — as in Mothers for Peace — an agency is headed by a multi-member board, the deliberations among those members are analogous to the internal mental processes of the sole head of an agency, and thus are generally not within the scope of the administrative record. Cf. Portland Audubon, 984 F.2d at 1549 (distinguishing Mothers for Peace as involving "the internal deliberative processes of the agency [and] the mental processes of individual agency members") (emphases added). No such deliberations among a multi-member agency are at issue here. The district court's decision to require a privilege log and evaluate claims of privilege on an individual basis before including documents in the record was not clearly erroneous as a matter of law.

* * *

The district court's October 17, 2017 order represents a reasonable approach to managing the conduct and exigencies of this important litigation — exigencies which were dictated by the government's March 5, 2018 termination date for DACA. In order for the government to prevail in its request for the extraordinary remedy of mandamus, we must be "left with a definite and firm conviction that a mistake has been committed." In re Bundy, 840 F.3d at 1041 (quoting In re United States, 791 F.3d at 955). We are left with no such conviction here, and mandamus relief is therefore not appropriate.

Accordingly, the stay of proceedings entered on October 24, 2017 is lifted.

PETITION DENIED.

WATFORD, Circuit Judge, dissenting:

I understand why the district court ordered the Department of Homeland Security (DHS) to provide a more fulsome administrative record. The agency's decision to rescind DACA will profoundly disrupt the lives of hundreds of thousands of people, and a policy shift of that magnitude presumably would not have been made without extensive study and analysis beforehand. But the desire for greater insight into how DHS arrived at its decision is not a legitimate basis for ordering the agency to expand the administrative record, unless the plaintiffs make a threshold factual showing justifying such action. They have not done so here. As a result, I think the district court's order constitutes "a clear abuse of discretion," and the burden imposed by the order is exceptional enough to warrant the extraordinary remedy of mandamus. Cheney v. United States District Court for the District of Columbia, 542 U.S. 367, 380, 124 S.Ct. 2576, 159 L.Ed.2d 459 (2004).

The district court's order violates two well-settled principles governing judicial review of agency action under the Administrative Procedure Act. The first is that a court ordinarily conducts its review "based on the record the agency presents to the reviewing court." Florida Power & Light Co. v. Lorion, 470 U.S. 729, 743-44, 105 S.Ct. 1598, 84 L.Ed.2d 643 (1985); see also Camp v. Pitts, 411 U.S. 138, 142, 93 S.Ct. 1241, 36 L.Ed.2d 106 (1973) (percuriam). If the record compiled by the agency is inadequate to support the challenged action, the reviewing court will usually be required to vacate the agency's action and remand for additional investigation or explanation. Florida Power, 470 U.S. at 744, 105 S.Ct. 1598. So in most cases the agency bears the risk associated with filing an incomplete record, not the challengers.

The second principle is that documents reflecting an agency's internal deliberative processes are ordinarily not part of the administrative record. See In re Subpoena Duces Tecum Served on the Office of the Comptroller of the Currency, 156 F.3d 1279, 1279-80 (D.C. Cir. 1998); Kansas State Network, Inc. v. FCC, 720 F.2d 185, 191, (D.C. Cir. 1983). An agency generally has no obligation to include documents that were prepared to assist the decision-maker in arriving at her decision, such as memos or emails containing opinions, recommendations, or advice. These pre-decisional materials are not deemed part of the administrative record because they are irrelevant to the reviewing court's task. The court's function is to assess the lawfulness of the agency's action based on the reasons offered by the agency, Motor Vehicle Manufacturers Association of the United States, Inc. v. State Farm Mutual Automobile Insurance Co., 463 U.S. 29, 50, 103 S.Ct. 2856, 77 L.Ed.2d 443 (1983), not to "probe the mental processes" of agency decision-makers in reaching their conclusions. Morgan v. United States, 304 U.S. 1, 18, 58 S.Ct. 773, 82 S.Ct. 1129 (1938). Requiring routine disclosure of deliberative process materials would also chill the frank discussions and debates that are necessary to craft well-considered policy. See Assembly of the State of California v. Department of Commerce, 968 F.2d 916, 920 (9th Cir. 1992); San Luis Obispo Mothers for Peace v. Nuclear Regulatory Commission, 789 F.2d 26, 45 (D.C. Cir. 1986) (en banc) (plurality opinion).

There are exceptions to these general rules. First, expansion of the record may be required when the agency fails to make formal findings and thus leaves the reviewing court unable to discern the agency's reasons for taking the action that it did. See Public Power Council v. Johnson, 674 F.2d 791, 793-94 (9th Cir. 1982). (This exception doesn't apply here because the memo issued by the Acting Secretary explicitly states her asserted reason for rescinding DACA: concern that the program would be invalidated in threatened litigation.) Second, the record may be expanded if there is evidence that the agency cherry-picked the materials it included by omitting factual information undermining the conclusions it reached. See Portland Audubon Society v. Endangered Species Committee, 984 F.2d 1534, 1548 (9th Cir. 1993). And third, documents reflecting an agency's internal deliberations may on occasion be made part of the record, but only if the challengers make "a strong showing of bad faith or improper behavior" on the part of agency decision-makers. Citizens to Preserve Overton Park, Inc. v. Volpe, 401 U.S. 402, 420, 91 S.Ct. 814, 28 L.Ed.2d 136 (1971), abrogated on other grounds by Califano v. Sanders, 430 U.S. 99, 97 S.Ct. 980, 51 L.Ed.2d 192 (1977); In re Subpoena Duces Tecum, 156 F.3d at 1279-80.

The plaintiffs have not made the showing necessary to trigger either of the latter two exceptions. They have not shown any likelihood that factual information considered by the Acting Secretary and relevant to her decision has been omitted from the record. Indeed, it would be implausible to think that any such material exists, given the nature of the reason asserted by the Acting Secretary for rescinding DACA. Concern over the program's vulnerability to legal challenge would rest not on factual information but on the legal analysis of lawyers. Documents analyzing DACA's potential legal infirmities, prepared to assist the Acting Secretary in assessing the gravity of the litigation risk involved, fall squarely within the category of deliberative process materials mentioned above. They are presumptively outside the scope of what must be included in the administrative record (and may be privileged in any event).

Nor have the plaintiffs attempted at this stage of the case to show bad faith or improper behavior on the part of the Acting Secretary. To be sure, they assert in their brief that they suspect her stated reason for rescinding DACA is pretextual. But bare assertions of that sort fall far short of the showing needed to overcome the presumption that agency decision-makers have acted for the reasons they've given.

Because the plaintiffs have failed to establish that any of these exceptions apply, I don't think the district court's order can stand. The court directed DHS to include in the administrative record all DACA-related "emails, letters, memoranda, notes, media items, opinions, and other materials" considered by the Acting Secretary, and all such materials considered by any other government official — including officials from the Department of Justice and the White House — who provided the Acting Secretary with written or verbal input on the decision to rescind DACA. The court further expanded the record to include "all comments and questions propounded by Acting Secretary Duke to advisors or subordinates or others regarding the actual or potential rescission of DACA and their responses."

In my view, the district court exceeded the scope of its lawful authority to expand the administrative record. The order sweeps far beyond materials related to the sole reason given for rescinding DACA — its supposed unlawfulness and vulnerability to legal challenge. The order requires the inclusion of all documents mentioning DACA-related issues of any sort, and is overbroad for that reason alone. But even if the order had been limited to documents analyzing the risk that DACA might be invalidated, those materials are deliberative in character and thus could not be made part of the administrative record absent a showing of bad faith or improper behavior. And to the extent the order will compel the production of communications between the Acting Secretary and high-level officials in the White House — including, potentially, the President himself — the order raises the same sensitive separation-of-powers concerns that made mandamus relief appropriate in Cheney. See 542 U.S. at 389-90, 124 S.Ct. 2576.

These departures from settled principles are enough to establish that the district court's order is "clearly erroneous as a matter of law," which is the most important of the factors we consider when deciding whether to grant mandamus relief. In re Bundy, 840 F.3d 1034, 1041 (9th Cir. 2016). The other factors weigh in favor of granting relief as well. The order isn't immediately appealable, and if relief is denied the harm inflicted will be immediate and irreparable. As the declarations submitted by the government attest, the search for documents responsive to the court's order will be burdensome and intrusive, given the large number of government officials who may have provided written or verbal input to the Acting Secretary. And the damage caused by public disclosure of otherwise privileged materials can't be undone following an appeal from the final judgment.

This strikes me as a classic case in which mandamus relief is warranted, and I would therefore grant the writ.