A Brief History of the PATRIOT Reauthorization Debate

I wanted to provide some background of how we got to this week’s PATRIOT Reauthorization debate to explain what I believe the surveillance boosters are really aiming for. Rather than a response to Edward Snowden, I think it is more useful to consider “reform” as an Intelligence Community effort to recreate functionalities they had and then lost in 2009.

2009 violations require NSA to start treating PATRIOT data like PATRIOT data and shut down automated functions

That history starts in 2009, when NSA was still operating under the system they had established under Stellar Wind while pretending to abide by FISC rules.

At the beginning of 2009, the NSA had probably close to full coverage of phone records in the US, and coverage on the most important Internet circuits as well. Contrary to the explicit orders of the FISC, NSA was treating all this data as EO 12333 data, not PATRIOT data.

On the Internet side, it was acquiring data that it considered Dialing, Routing, Addressing, and Signaling information but which also constituted content (and which violated the category limits Colleen Kollar-Kotelly had first imposed).

On the phone side, NSA was not only treating PATRIOT data according to NSA’s more general minimization procedures as opposed to those dictated by the FISC. But in violation of those minimization procedures, NSA was submitting phone dragnet data to all the automated procedures it submitted EO 12333 data to, which included automated searches and automatic chaining on other identifiers believed to belong to the same user  (the latter of which NSA calls “correlations”). Either these procedures consisted of — or the data was also treated to — pattern analysis, chaining users on patterns rather than calls made. Of key importance, one point of having all the data in the country was to be able to run this pattern analysis. Until 2008 (and really until 2009) they were sharing the results of this data in real time.

Having both types of data allowed the NSA to chain across both telephony and Internet data (obtained under a range of authorities) in the same query, which would give them a pretty comprehensive picture of all the communications a target was engaging in, regardless of medium.

I believe this bucolic state is where the surveillance hawks want us to return to. Indeed, to a large extent that’s what Richard Burr’s bill does (with a lot of obstructive measures to make sure this process never gets exposed again).

But when DOJ disclosed the phone violations to FISC in early 2009, they shut down all those automatic processes. And Judge Reggie Walton took over 6 months before he’d even let NSA have full ability to query the data.

Then, probably in October 2009, DOJ finally confessed to FISC that every single record NSA had collected under the Internet dragnet for five years violated Kollar-Kotelly’s category rules. Walton probably shut down the dragnet on October 30, 2009, and it remained shut down until around July 2010.

At this point, not only didn’t NSA have domestic coverage that included Internet and phone, but the phone dragnet was a lot less useful than all the other phone data NSA collected because NSA couldn’t use its nifty automatic tools on it.

Attempts to restore the pre-2009 state

We know that NSA convinced John Bates to not only turn the Internet dragnet back on around July 2010 (though it took a while before they actually turned it on), but to expand collection to some or all circuits in the US. He permitted that by interpreting anything that might be Dialing, Routing, Addressing, and Signaling (DRAS) to be metadata, regardless of whether it also was content, and by pointing back to the phone dragnet to justify the extension of the Internet dragnet. Bates’ fix was short-lived, however, because by 2011, NSA shut down that dragnet. I wildarseguess that may partly because DOJ knew it was still collecting content, and when Bates told NSA if it knew it was collecting content with upstream collection, it would be illegal (NSA destroyed the Internet dragnet data at the same time it decided to start destroying its illegal upstream data). I also think there may have been a problem with Bates’ redefinition of DRAS, because Richard Burr explicitly adopted Bates’ definition in his bill, which would have given Bates’ 2010 opinion congressional sanction. As far as we know, NSA has been coping without the domestic Internet dragnet by collecting on US person Internet data overseas, as well as off PRISM targets.

Remember, any residual problems the Internet dragnet had may have affected NSA’s ability to collect any IP-based calls or at least messaging.

Meanwhile, NSA was trying to replace the automated functions it had up until 2009, and on November 8, 2012, the NSA finally authorized a way to do that. But over the next year plus, NSA never managed to turn it on.

The phone records gap

Meanwhile, the phone dragnet was collecting less and less of the data out there. My current theory is that the gap arose because of two things involving Verizon. First, in 2009, part or all of Verizon dropped its contract with the FBI to provide enhanced call records first set up in 2002. This meant it no longer had all its data collected in a way that was useful to FBI that it could use to provide CDRs (though Verizon had already changed the way it complied with phone records in 2007, which had, by itself, created some technical issues). In addition, I suspect that as Verizon moved to 4G technology it didn’t keep the same kind of records for 4G calls that transited its backbone (which is where the records come from, not from customer bills). The problems with the Internet dragnet may have exacerbated this (and in any case, the phone dragnet orders only ask for telephony metadata, not IP metadata).

Once you lose cell calls transiting Verizon’s backbone, you’ve got a big hole in the system.

At the same time, more and more people (and, disproportionately, terrorist targets) were relying more and more on IP-based communications — Skype, especially, but also texting and other VOIP calls. And while AT&T gets some of what crosses its backbone (and had and still has a contract for that enhanced call record service with the FBI, which means it will be accessible), a lot of that would not be available as telephony. Again, any limits on Internet collection may also impact IP based calls and messaging.

Edward Snowden provides a convenient excuse

Which brings you to where the dragnets were in 2013, when Edward Snowden alerted us to their presence. The domestic PATRIOT-authorized Internet dragnet had been shut down (and with it, potentially, Internet-based calls and messaging). The phone dragnet still operated, but there were significant gaps in what the telecoms would or could turn over (though I suspect NSA still has full coverage of data that transits AT&T’s backbone). And that data couldn’t be subjected to all the nifty kinds of analysis NSA liked to subject call data to. Plus, complying with the FISC-imposed minimization procedures meant NSA could only share query results in limited situations and even then with some bureaucratic limits. Finally, it could only be used for counterterrorism programs, and such data analysis had become a critical part of all of NSA’s analysis, even including US collection.

And this is where I suspect all those stories about NSA already considering, in 2009 and in 2013, shutting down the dragnet. As both Ken Dilanian stories on this make clear, DOJ believed they could not achieve the same search results without a new law passed by Congress. Bob Litt has said the same publicly. Which makes it clear these are not plain old phone records.

So while Edward Snowden was a huge pain in the ass for the IC, he also provided the impetus to make a decision on the phone dragnet. Obama made a big show of listening to his Presidential Review Group and PCLOB, both of which said to get rid of it (the latter of which said it was not authorized by Section 215). But — as I noted at the time — moving to providers would fix some of their problems.

In their ideal world, here’s what we know the IC would like:

  • Full coverage on both telephony and IP-based calls and messaging and — ideally — other kinds of Internet communications
  • Ability to share promiscuously
  • Ability to use all NSA’s analytical tools on raw data (the data mandates are about requiring some kind of analytical work from providers)
  • Permission to use the “call” function for all intelligence purposes
  • Ability to federate queries with data collected under other authorities

And the IC wants this while retaining Section 215’s use of bulky collections that can be cross-referenced with other data, especially the other Internet collection it conducts using Section 215, which makes up a majority of Section 215 orders.

Those 5 categories are how I’ve been analyzing the various solutions (which is one of about 10 reasons I’m so certain that Mitch McConnell would never want straight reauthorization, because there’s nothing that straight reauthorization would have ratified that would have fixed the existing problems with the dragnet), while keeping in mind that as currently constructed, the Internet 215 collection is far more important to the IC than the phone dragnet.

How the bills stack up

USA F-ReDux, as currently incarnated, would vastly expand data sharing, because data would come in through FBI (as PRISM data does) and FBI metadata rules are very permissive. And it would give collection on telephony and IP-based calls (probably not from all entities, but probably from Apple, Google, and Microsoft). It would not permit use for all intelligence purposes. And it is unclear how many of NSA’s analytical tools they’d be able to use (I believe they’d have access to the “correlations” function directly, because providers would have access internally to customers’ other accounts, but with the House report, other kinds of analysis should be prohibited, though who knows what AT&T and Microsoft would do with immunity). The House report clearly envisions federated queries, but they would be awkward to integrate with the outsourced collection.

Burr’s bill, on the other hand, would expand provider based querying to all intelligence uses. But even before querying might —  maybe — probably wouldn’t — move to providers in 2 years, Burr’s bill would have immediately permitted NSA to obtain all the things they’d need to return to the 2009 bucolic era where US collected data had the same treatment as EO 12333 collected data. And Burr’s bill would probably permit federated queries with all other NSA data. This is why, I think, he adopted EO 12333 minimization procedures, which are far more restrictive than what will happen when data comes in via FBI, because since it will continue to come in in bulk, it needs to have an NSA minimization procedure. Burr’s bill would also sneak the Section 215 Internet collection back into NSL production, making that data more promiscuously available as well.

In other words, this is why so many hawks in the House are happy to have USA F-ReDux: because it is vastly better than the status quo. But it’s also why so many hawks in the Senate are unsatisfied with it: because it doesn’t let the IC do the other things — some of the analytical work and easy federated queries — that they’d like, across all intelligence functions. (Ironically, that means even while they’re squawking about ISIS, the capabilities they’d really like under Burr’s bill involve entirely other kinds of targets.)

A lot of the debate about a phone dragnet fix has focused on other aspects of the bill — on transparency and reporting and so on. And while I think those things do matter (the IC clearly wants to minimize those extras, and had gutted many of them even in last year’s bill), what really matters are those 5 functionalities.


Marcy has been blogging full time since 2007. She’s known for her live-blogging of the Scooter Libby trial, her discovery of the number of times Khalid Sheikh Mohammed was waterboarded, and generally for her weedy analysis of document dumps.

Marcy Wheeler is an independent journalist writing about national security and civil liberties. She writes as emptywheel at her eponymous blog, publishes at outlets including the Guardian, Salon, and the Progressive, and appears frequently on television and radio. She is the author of Anatomy of Deceit, a primer on the CIA leak investigation, and liveblogged the Scooter Libby trial.

Marcy has a PhD from the University of Michigan, where she researched the “feuilleton,” a short conversational newspaper form that has proven important in times of heightened censorship. Before and after her time in academics, Marcy provided documentation consulting for corporations in the auto, tech, and energy industries. She lives with her spouse and dog in Grand Rapids, MI.

7 replies
  1. wallace says:

    quote”In their ideal world, here’s what we know the IC would like:”unquote

    The abolition of the Bill of Rights, and the rest of the Constitution, which would be replaced by a Totalitarian constitution granting the Executive legal imperialism powers unseen since Henry 8th and the IC powers of surveillance and militarized law enforcement imbued with Burrs idea of Tyrants-R-Us powers to prosecute those who would dare expose their tyrannical freefall into Fascism on steroids. Benito Mussolini would pin a medal of honor on Burr and McConnell. I’d place a bullet in their temples. The Framers would simply hang them.

    • wallace says:

      quote”The Framers would simply hang them.”unquote

      And then again, some who’ve been dragged through hell because of scumbag US Attorney tyrants of the DOJ just might drag them out of their granite palaces..hang them from the closest lamp post upside down..draw and quarter them ..and burn their entrails in the street. After all..Piketty already warned them.

      • wallace says:

        quote”And then again, some who’ve been dragged through hell because of scumbag US Attorney tyrants of the DOJ “unquote

        A few survivors of Waco and Ruby Ridge come to mind. Amongst 1000’s of those who are now serving long time in Murika’s private prison complex for “drug crimes” deemed by the insidious Two Tiered justice as worthy of life in prison while pond scum cockroaches like Patreaus are slapped on the wrist with “misdemeanors” of divulging classified info to their scumsucking lover.

        At some point, if this whole fucking misplacement of “justice” in this country, notwithstanding the insidious deep state surveillance being quantified in our Congress isn’t enough to to eventually cause the citizens of this country to draw a physical armed line in the sand..then they deserve to fall into the abyss. I just feel sorry for our future generations..because.. once THEY understand what our generation cowardly failed to do..THEY will do it..and then spit on our graves.

  2. jerryy says:

    Normally I do not do this sort of sharing of messages the congress critters send me, but I am going to do so this time. Senator McConnell has indeed sent to me another missive designed to assure me he is right in his pursuits and that those who do not see it his way are mistaken:
    The relevant parts from his letter to me (quoted just as he or his staffer wrote it):
    …”While I will certainly keep your thoughts in mind, I seek to clarify some common misconceptions pertaining to the extensions. For your convenience, I have listed below the implications that each of the provisions at-issue will have on U.S. citizens.
    One provision allows our federal intelligence community, with a federal court order, to continue to monitor particular terror suspects who use multiple telephone numbers or e-mail addresses for terror related communications.  Ruled constitutional by the courts, this provision is crucial to our intelligence agencies for tracking the movements of suspected terrorists, who otherwise may evade detection through the use of tactics such as “throw away” cellular phones.
    A second provision permits the federal government to obtain business records already in the possession of a third party but only after a federal court finds reasonable grounds to believe that such records are relevant to a foreign intelligence or international terrorism investigation.  The federal government, however, is not permitted to seek such an order related to an American based solely on activities protected by the First Amendment.
    The third and final provision allows the federal government, again upon the issuance of a federal court order, to investigate suspected foreign terrorists who have no apparent ties to a foreign power or identified terror network. Known as the so-called “lone wolf” provision, the law explicitly prohibits the use of this provision to investigate American citizens or legal permanent residents.
    Today we face a diffuse and versatile threat from international terrorist groups like ISIL and we need every tool available to combat these organizations intent on striking America and its allies.   We, as a nation, must remain vigilant in combating the threat posed by terror both at home and abroad.  That is why it is so imperative that law enforcement continue to have key investigative tools, like the three provisions previously mentioned, in order to continue their efforts to keep our nation safe from the threat posed by domestic and international terrorists.”

    Who knows, maybe he even believes it. Nahhhh…

  3. orionATL says:

    the central question for history and politics – though not for any immediate action needed – is what could possibly motivate mcconnell and burr to this extent ? what is either going to get out of this ? what makes either think that in 2020 an opponent won’t hang what they have done around their neck?

  4. A Reader says:


    ” …”While I will certainly keep your thoughts in mind, I seek to clarify some common misconceptions pertaining to the extensions. For your convenience, I have listed below the implications that each of the provisions at-issue will have on U.S. citizens.”

    I think the biggest problem these people have is the either willful or ignorant/naive inability to separate the chocolate from the peanut butter — or maybe the individual ingredients of any sandwich or culinary concoction in general.

    So, they think you’ll buy that they need to have that big massive grinder when you really might just want pickles. Maybe pickles are the only thing you can even tolerate on the entire grinder. Maybe you’re viciously allergic to several of the ingredients in the grinder. But to get the pickles they make you swallow the whole damn grinder. They prohibit you from getting pickles any other way.

    So you’re stuck with something deadly to get something you want. Which of course neglects the fact that you don’t really even NEED pickles, or the mammoth [calorie/rhetoric] bomb of that monstrous grinder, at all — and in fact that it can kill you and everything that matters to you.

    Messy example but I think it mostly gets my point across?

Comments are closed.