NSA Twice Chose to Forgo Privacy Protections in Domestic Data Mining Programs

While Jane Mayer’s profile on NSA whistleblower Thomas Drake has generated a lot of attention for the way Obama’s DOJ is senselessly prosecuting him, there has been less focus on the key revelation that Drake and others went on the record to reveal in Mayer’s story: that the NSA chose not to integrate the privacy protections from a program called ThinThread into its illegal domestic surveillance program.

Pilot tests of ThinThread proved almost too successful, according to a former intelligence expert who analyzed it. “It was nearly perfect,” the official says. “But it processed such a large amount of data that it picked up more Americans than the other systems.” Though ThinThread was intended to intercept foreign communications, it continued documenting signals when a trail crossed into the U.S. This was a big problem: federal law forbade the monitoring of domestic communications without a court warrant. And a warrant couldn’t be issued without probable cause and a known suspect. In order to comply with the law, [Bill Binney, a crypto-mathmetician who headed Signals Intelligence Automation Research Center (SARC) that developed ThinThread] installed privacy controls and added an “anonymizing feature,” so that all American communications would be encrypted until a warrant was issued. The system would indicate when a pattern looked suspicious enough to justify a warrant.


When Binney heard the rumors, he was convinced that the new domestic-surveillance program employed components of ThinThread: a bastardized version, stripped of privacy controls. “It was my brainchild,” he said. “But they removed the protections, the anonymization process. When you remove that, you can target anyone.” He said that although he was not “read in” to the new secret surveillance program, “my people were brought in, and they told me, ‘Can you believe they’re doing this? They’re getting billing records on U.S. citizens! They’re putting pen registers’ ”—logs of dialled phone numbers—“ ‘on everyone in the country!’ ”


[Former HPSCI staffer Diane Roark] asked Hayden why the N.S.A. had chosen not to include privacy protections for Americans. She says that he “kept not answering. Finally, he mumbled, and looked down, and said, ‘We didn’t need them. We had the power.’ He didn’t even look me in the eye. I was flabbergasted.” She asked him directly if the government was getting warrants for domestic surveillance, and he admitted that it was not. [my emphasis]

Mayer’s actually not the first to report on the decision not to implement the privacy protections of ThinThread. It was the subject of one of Siobhan Gorman’s articles during the period when Drake, according to the indictment, served as a source for her. The article appeared on May 18, 2006, the morning of Michael Hayden’s confirmation hearing to be CIA Director. (Unlike most of Gorman’s articles from the period, this appears to be available only behind the Sun’s firewall. Update: I’ve found a link to the article at CommonDreams.) It describes that since Bush’s authorization for the program required no privacy protections, the NSA just didn’t bother to implement that part of ThinThread.

Once President Bush gave the go-ahead for the NSA to secretly gather and analyze domestic phone records – an authorization that carried no stipulations about identity protection – agency officials regarded the encryption as an unnecessary step and rejected it, according to two intelligence officials knowledgeable about ThinThread and the warrantless surveillance programs.”They basically just disabled the [privacy] safeguards,” said one intelligence official.

A former top intelligence official said that without a privacy requirement, “there was no reason to go back to something that was perhaps more difficult to implement.”

However two officials familiar with the program said the encryption feature would have been simple to implement. One said the time required would have involved minutes, not hours. [my emphasis; bracket original]

In other words, ThinThread came equipped with a measure–encryption–to achieve the same thing as minimization, but before the fact. But in implementing Dick Cheney’s illegal wiretapping, NSA took that protection out of the program. And when asked why he had done that, Michael Hayden explained they didn’t need the protection, not with the Presidential authorization they used to justify the program.

October 2001, as Michael Hayden was implementing Cheney’s illegal program, was not the only time the government chose not to include privacy protections on a data mining program focused on Americans.

As Shane Harris reported in 2006 and in more detail in his book, The Watchers, when the government dismantled John Poindexter’s Total Information Awareness program in August 2003 after Congress defunded it, they didn’t actual dismantle most of it–they just moved it into the NSA. In his book, Harris described Poindexter’s regret that the government had not salvaged the privacy protection research.

But he regretted that the privacy research had been tossed into the dustbin. He’d never felt that the idea got traction, and what little research there’d been would wither without funding. It was a fateful decision, since the agency inheriting TIA would so on enough find itself accused of a massive and illegal incursion into Americans’ private lives.

So in October 2001, NSA affirmatively chose to disable privacy protections in ThinThread, and then again in August to December 2003, the government chose to salvage the data mining aspects of Total Information Awareness, but not the privacy research.

In other words, the government, on at least two occasions, chose not to incorporate existing technology into its data mining program to protect the privacy of Americans. Sort of makes it clear that the Bush Administration wanted to make sure Americans’ privacy wasn’t protected, huh?


  1. lysias says:

    Eavesdropping played a critical role in the establishment of the Nazi dictatorship in Germany. It was carried out by one of the least well-known but most important intelligence agencies, the Forschungsamt, established within Hermann Göring’s Air Ministry.

    It was officially established on Apr. 10, 1933, by which point Göring had gotten the additional title of Air Minister. However, Göring was already Reich Air Commissar on Feb. 2, and he was Interior Minister of Prussia, supervising the largest police force within Germany, once Hitler’s coalition took office on Jan. 30. It was the police role that made it possible for Göring to organize, as he probably did, the Reichstag Fire on Feb. 27.

    But eavesdropping played a critical role before the Forschungsamt was officially established. At the beginning of 1933, Ernst Oberfohren was head of the Reichstag fraction of the German Nationalist Party (DNVP), the junior partners in Hitler’s coalition. Oberfohren opposed his party chief Hugenberg’s policy of close cooperation with the Nazis, and he was trying to stir up opposition to that policy among fellow members of his party. On Mar. 29, 1933, just days after Hitler’s Enabling Act had been passed, Oberfohren’s offices in Berlin and in his home town of Kiel were searched by the police, after his telephone conversations had been eavesdropped upon. Oberfohren was forced to resign his offices the next day, and on May 8 he was the victim of a suspicious suicide, shortly before his party, the DNVP, committed suicide itself by dissolving itself on June 27.

  2. Teddy Partridge says:

    Michael Hayden should be in prison.

    Reading Mayer’s article last night, this was exactly what I’d hope you would focus on, Marcy. It’s not as if they designed a system but forgot the restrictions on surveilling US persons. NSA actually took a system with those affirmative protections and stripped them out.

    Of course, when the Vice President does it, it must be legal. I particularly liked the irony that Hill staffer Roark went to David Addington with her concerns about the program. Must have been a low chuckle or two around the man-sized safe when that happened.

  3. Deep Harm says:

    Every career federal position has a set of performance elements that provide the basis for determining whether that employee is performing adequately. One or more of the performance elements is labeled “critical,” meaning that, regardless of performance on all other elements, the employee must perform adequately on the critical elements in order to remain in the position. The President has just one critical performance element…expressed in the oath of office.

    I do solemnly swear (or affirm) that I will faithfully execute the Office of President of the United States, and will to the best of my ability, preserve, protect and defend the Constitution of the United States.

    Therefore, protecting the Constitution must take precedence over all other duties of the President, even activities that involve national security, for the oath makes no exceptions for anything else. Yet, the last two Presidents have routinely subordinated provisions of the Constitution to secrecy, warmaking, and other considerations. Arguably, those failures would be grounds for removal, i.e., impeachment.

  4. Jeff Kaye says:

    From EFF, 5/20/11

    Washington, D.C. – The Electronic Frontier Foundation (EFF) has filed a Freedom of Information Act (FOIA) suit against the Department of Justice (DOJ), demanding the release of a secret legal memo used to justify FBI access to Americans’ telephone records without any legal process or oversight.

    A report released last year by the DOJ’s own Inspector General revealed how the FBI, in defending its past violations of the Electronic Privacy Communications Act (ECPA), had come up with a new legal argument to justify secret, unchecked access to private telephone records. According to the report, the DOJ’s Office of the Legal Counsel (OLC) had issued a legal opinion agreeing with the FBI’s theory. That legal opinion is the target of the FOIA lawsuit filed Thursday….

    Earlier this year, the DOJ denied a FOIA request from a journalist seeking disclosure of the secret OLC opinion and in doing so revealed — perhaps inadvertently — the specific portion of the law on which the FBI’s aggressive legal theory relies. Based on its analysis of that particular ECPA provision, 18 U.S.C. 2511(2)(f), EFF fears that the FBI and OLC have wrongly concluded that national security investigators are free to obtain records of Americans’ international communications without first obtaining a subpoena or any other legal process. With this additional information about the contents of the OLC opinion, EFF filed its own FOIA request, but the DOJ has continued to stall its release….

    For the full FOIA lawsuit:

  5. Scarecrow says:

    And should we assume these measures without the privqcy features are still in use? Or did someone find a copy of the Fourth Amendment in Cheneys safe and actually read it?

    • nahant says:

      Nope they burned the evidence I am sure. Remember that convenient little fire in Cheny’s office area(Dec 19, 2007 … Thick black smoke billowed from a fire Wednesday on the White House compound in the Eisenhower Executive Office Building.).. Gee I wonder just went up in smoke…

      • onitgoes says:

        Gee I wonder just went up in smoke…

        I realize your “question” was rhetorical, but I have an answer. What went up in smoke with some odd little fire at Darth Cheney’s crypt?

        1. The US Constitution (what was left of it after being used as toilet paper for so long)

        2. The Bill of Rights (ditto)

        3. The Rule of Law (teh funneeee, tee hee, “rule” of “law” rotflol!!!)

        4. Civil rights (ditto), esp the right to privacy (guffaw guffaw snort)

        5. The concept of “Innocent until proven guilty” (uproarious laughter)

        6. Your right to a speedy trial (hysterical laughter)

        7. Your right to adequate representation in a court of “law” (guffaw)

        8. Freedom (insane laughter)

        9. A social safety net for all taxpaying US citizens (har de harharhar)

        10. Dignity (whad-dat??)

        11. Respect for the sanctity of human life (only applies when fooling the Teahadists that conservative pols actually *care* about the “rights” of a fetus… har har har har har snort guffaw… DUMBOS)

        12. Checks & Balances among the three branches of US Govt (go ask the Koch Brothers about that when they stop pissing their pants laughing so hard all the way to their numerous off-shore bank accounts…)

        There may be something I’ve missed, but I think that’s a reasonably comprehensive list of what got TRASHED by Cheney.

    • emptywheel says:

      We should assume that.

      One of the things Mike McConnell consistently objected to in PAA and FAA negotiations was oversight over minimization.

      I’ll lay this out in more detail in a follow-up post. But the debate about minimization, it seems, has always been about using this information for other purposes.

  6. MadDog says:

    Just so folks understand, the definition that our government uses defines data mining as not data mining (page 2 of 10 page PDF):

    …The Data Mining Reporting Act requires “the head of each department or agency of the Federal Government that is engaged in an activity to use or develop data mining shall submit a report to Congress on all such activities of the department or agency.”2 This is an annual requirement.

    Under the Act, “data mining” is defined as: “… a program involving pattern-based queries, searches or other analyses of one or more electronic databases, where —

    (A) a department or agency of the Federal Government, or a non-Federal entity acting on behalf of the Federal Government, is conducting the queries, searches, or other analyses to discover or locate a predictive pattern or anomaly indicative of terrorist or criminal activity on the part of any individual or individuals;

    (B) the queries, searches, or other analyses are not subject-based and do not use personal identifiers of a specific individual, or inputs associated with a specific individual or group of individuals, to retrieve information from the database or databases; and

    (C) the purpose of the queries, searches, or other analyses is not solely— (i) the detection of fraud, waste, or abuse in a Government agency or program; or (ii) the security of a Government computer system.3”

    (My Bold)

    • papau says:

      So data mining privacy security is not in question if national security is the topic? Sort of like waterboarding is not torture if no major organs are permanently harmed.

      • MadDog says:

        From what I can tell, it’s even worse.

        More like “waterboarding is not torture if no major organs are permanently harmed…and even if major organs are harmed, it’s not torture just because we say so.”

    • michaelfishman says:

      According to your link, the ODNI memo says that the Data Mining Reporting Act

      defines data mining as you indicate, and that the act implements the Recommendations of the 9/11 Commission Act.

      Do you have any idea how Congress was led to define data mining so narrowly as to give NSA such a free hand?

  7. orionATL says:

    the privacy safeguards would have been ignored with intent, that is to say, with the intention that the administration would be able to eavesdrop on selected individuals without violating any administrative rules –

    journalists for sure, politicians likely, soldiers for sure, diplomats absolutely, doubtful allies for sure, anti-war activists of any stripe for sure.

    oh, and maybe a terrorista here and there -for practice.

    for the most of us, the bush admin could have cared less whether we did or did not have our privacy rights violated; we simply were of no interest to them.

    nor was “national security” of great interest to the bush admin.

    the main interest, i would bet, was focused primarily on those who could impede the march to invasion and to war, and then those who could effectively criticize the war and its conduct.

  8. orionATL says:

    oh, a question arises:

    “…Sort of makes it clear that the Bush Administration wanted to make sure Americans’ privacy wasn’t protected, huh?…”

    what has the obama admin done, if anything, to rescind this failure of the bush admin to choose and use available safeguards?

  9. PeasantParty says:

    I wonder if our Congress and Military folks ever think that these vehicles to spy on us might in fact be doing the same to them. You know, it would actually be some seriously sweet justice if a few phone secrets were to fly out into the open.

  10. PeasantParty says:

    Come to think of it, we’ve been paying full price for private phone lines for years. I think the telecos owe us some money back for forcing us to use only party lines.

  11. robspierre says:

    “Sort of makes it clear that the Bush Administration wanted to make sure Americans’ privacy wasn’t protected.”

    Exactly. And not just the Bushistas, either. The next question is “why”?

    As I mentioned in another thread just a minute ago, privacy protections for Americans also improve the efficiency of antiterrorist/foreign-intelligence surveillance by filtering and focusing the search. Better focus reduces the false-positive rate, saves time, and improves data quality. So eliminating these features runs counter to the stated purposes of the program.

    Which suggests that the stated prupose is not the real purpose. In this country, surveillance no longer has anything much to do with security or fighting crime. It is a tool for intimidating the populace, dominating the political discourse, and, quite likely, for out-and-out, private-venture criminality, such as stock-market manipulation.