A Guide to Domestic Surveillance

Jane Harman is confused. In her statement responding to yesterday’s court decision, she said (looking for a link, not up on her site yet):

Today, a federal court in Detroitstruck down as unconstitutional the President’s NSA Program.  Thedecision is significant in that it represents the first judicialdetermination that the President’s program violates the law and theConstitution.

The terrorists who are plotting against us would like nothing more thanto see us erode our Constitution.  We cannot hand them the victory theyseek.  For that reason, it is essential that all electronicsurveillance of Americans comply with the Foreign IntelligenceSurveillance Act and the Constitution.

By failing to create a legislative framework for this program, theAdministration and Congress have punted this matter to the courts. Nobody should be surprised that the ensuing litigation has createdinconsistent results in different courts and will only contribute tothe "fog of law" that has surrounded this program.

She refers to inconsistent results–which is misleading. You see, the two main cases on the warrantless wiretapping program have not yet shown inconsistent results. Harman appears to be conflating the underlying data collection program (which is probably illegal) with the eavesdropping program (which is definitely illegal).

Of course, Harman is not alone. Bloggers and reporters are also conflating the two programs (though to be fair, unlike Harman, the bloggers and reporters haven’t been briefed on these programs repeatedly). So, as a service to readers, I’m going to post my understanding of the different aspects of what Bush’s NSA is doing to Americans. Of course, this is my understanding–please provide corrections where you’ve got them.

  1. Anonymous says:

    I’m working on some updates to this, but in the meantime, here’s a great post suggesting what the original logic of hte data collection program was. Apparently, AT&T didn’t adequately hide the following claims made in the Hepting case;

    â€AT&T notes that the facts recited by plaintiffs are entirely consistent with any number of legitimate Internet monitoring systems, such as those used to detect viruses and stop hackers,†the redacted pages say.

    In other words, I think that’s what the original justification for letting the NSA into the circuits (it does have a defensive role like this, and it is probably necessarily from a technical standpoint).

  2. kim says:

    I’ve wondered how the NSA can both coordinate and monitor massive amounts of phone and web data from widely distributed locations, without using the/a web in some way?

  3. Anonymous says:


    They’re collecting the data from the circuits, from the roughly ten places in the country that all data have to physically pass through. So it’s not like they have to go to different places to collect the data. Also, they’ve got a contractor helping with the monitoring process, using some fancy software.

  4. DeanOR says:

    I’m sure it’s just an oversight, but I think the chart is inaccurate where it says â€Wiretap on people identified through data mining (probably based on profiles developed from captured Al Qaeda operatives), without first getting a warrant in the FISA court.†The problem is the word â€firstâ€. Bush supporters say that it is too cumbersome and slow to â€first†get a warrant from FISA when dealing with terrorists, which they use as a justification for violating FISA. But I believe FISA allows wiretapping before actually getting the warrant. The warrant can then be obtained within a specified time period after the wiretapping begins. So that partifular BushCo justification for violating the law is invalid.

  5. margaret says:

    Golly, Moses! This conjures up the image of millions of grimlins hovering over computers to catch names. Surely, even if the computers sift out all the chaff, there is too much â€wheat†for analysts to â€eat.†And, how many of these grimlims speak Arabic?

    Considering how inefficient government bureaucracy is, I haven’t much fear about it, in general. However, the creepiness of it all
    is offensive. Data pornography.

  6. Anonymous says:


    They’re not collecting names or sorting for words. The data collection part (and probably the data mining part) are looking at call data–information about who you called and how long you spoke to them, not infomration about what you said. All that data is thrown into a database to try to discern particular patterns. If they were doing data mining before 9/11, they were just kind of doing it, without a target in mind. Afterwards, they undoubtedly started looking more closely such as the call patterns of those who make calls to Pakistan or Afghanistan.

    Dean OR

    Oh yeah, good point. Will update accordingly.

  7. kim says:

    They’re also collecting overseas (the major responsibility of the NSA it still overseas I hope!), as well as in the US… they have to store the raw data (not just connection info) before analysing it, they also have to be able to integrate all locations, then be able to data mine with changing criteria, then probably search for specific words or voices… I guess all this happens after bringing it all back to MD. Sometimes it seems like they’d need an entirely separate second â€shadow†net to accomplish this.

  8. Anonymous says:

    kim, two corrections.

    First, one of the two legitimate justifications for the telecoms giving the NSA access to the circuits is overseas wiretapping. A high percentage of the world’s communication comes through US circuits. So if you want to digitally wiretap, the best places to do so are SF and NJ, in the AT&T circuit room. (The other legitimate reason is to have a presence on the digital backbones to be able to pursue cyberhackers.)

    Second, this data collection program isn’t tied to names and words. From everything reasonably respectable we’ve seen on this, it’s not a â€seek for use of falafel†program. They do that after they’ve either identified someone through cause or using the data mining. The kind of searches they would do here are: â€seek for people who contact regularly with Afghanistan, regularly contact falafel delivery joints, and use travelocity.â€

  9. William Ockham says:


    I suspect that your taxonomy is incorrect.

    First, the data vacuuming (as described by Slate and USA Today) is pointless if there is not an associated data mining effort. Actually, calling this data mining is somewhat misleading (in the same way that there are no wires involved in the â€wiretappingâ€). What we’re really talking about is a sophisticated pattern recognition system that differs in some important ways from what â€data mining†has traditionally meant in software circles, but probably on real geeks care about that.

    Second, there is a scenario that would make the pre 9-11 data collection efforts legal and legitimate. Unfortunately, we’ll probably never know for sure. On the other hand, we can be pretty sure that the way the program was used after 9-11 was illegal, unconstitutional, and absolutely ineffective. I don’t have time to back up that assertion now, but if anybody seriously disagrees, I’ll get back to it later.

    Third, if the warrantless wiretapping is illegal and unconstitutional, the data collection effort is assuredly illegal (and probably unconstitutional, but that’s not clear to me). Even after the â€Patriot Act†changes, collecting call detail records still requires identifying the specific individual that the government is interested in. This is the basis for my response to Kagro X’s post. I don’t think it matters that the judge accepted the â€state secrets†claim with respect to the data mining. If her decision stands, the NSA will shut down the data collection effort.

    Fourth, based on what Alberto Gonzales refused to say at the Senate Judiciary Committee hearing, what Hayden said in his defense of the program and what Jay Rockefeller implied in his â€memo to fileâ€, I believe that the Administration is attempting to foist off a false distinction. After 9-11, the NSA merged these two programs into a single effort. The data collection effort doesn’t just involve internet traffic, the wiretapping isn’t just about voice communications, and the patterns aren’t based on analysis of terrorists’ calling patterns. as difficult as I find to say this, I think Harman is right about her description of the program. I wish she was more helpful in seeing this for what it really is: An authoritarian abuse of our civil liberties.

    To sum up, although the data collection started earlier than the warrantless wiretapping, the wiretapping should be seen as an outgrowth (and a particularly malignant one at that) of the earlier effort.

  10. Libby Sosume says:

    Though we may not like the sound of it, â€data mining†isn’t per se illegal. It’s the collection of the data without warrant that likely is illegal. People need to understand that simple point.

  11. Anonymous says:

    WO, I agree with everything you say, particularly your distinction between data mining and pattern recognition. I suspect they’ve used the word â€data mining†as a bit of a red herring that is more familiar to people. But I’ve tried to make this simple, at the risk of losing the nuance you’re pointing to, and for that reason I chose the language both the judges and the press are using to refer to this part of the program.

    But I don’t think that challenges my taxonomy itself. The laws are still different, and the basis for court challenges (both potential and actual, what we’ve seen) are different. You and I have standing in a class action case against AT&T. It’d be a lot harder to argue we have standing for a case against the NSA on the illegal wiretapping. This distinction is important to make, not least because there are different laws at stake. We know, for sure, that the illegal wiretaps violate FISA. But the AT&T case is more ambiguous.

    And I suspect there are actually two different factors that would be more accurate to include. First, there’s the presence of NSA on the network. Its cybersecurity function alone would justify this, and it would also allow the telecoms to claim a proper business function (that is, to access pen register data for the normal functioning of their business–if they say they’re doing it to prevent hackers, they can at least argue the case; as the first comment I made points out, that appears to be what they were prepared to do). Furthermore, NSA’s mandate to spy internationally requires that it have access to the circuits–you can’t tap using traditional analog methods anymore and presence in the circuits is the only thing that’ll give you that access and the only place to get it is here in the US.

    Then there’s the collection of all data, domestic and international. This is where you start violating Electronic Communications Privacy Act because you’re collecting domestic as well as international traffic (FWIW, one of Lichtblau’s and Risen’s articles included a quote that suggested this was the beginning of the end, the technical impossibility to separate international and domestic data).

    Then there’s the data mining (pattern recognition). The reason this is important, in this program, because no one has ever made a legal case for the validity of pattern recognition as a justification for probable cause. BushCo had the option to try, and thereby to continue this surveillance under FISA. But they didn’t do so, apparently because they were warned (I think by the FISA judges) it wouldn’t fly.

    And then there’s the wiretapping. Again, it’s legally distinct, because this is the first time they’re going beyond call data to call content (ambiguities about whether Internet â€call dataâ€â€“things like email headers and web searches) can be considered just register data aside).

    And I do’t see how what you say makes Jane Harman correct. There are two cases challenging the illegal wiretapping program. One has declared it illegal, the other hasn’t yet ruled. All the other decisions are irrelevant to this program. The illegal wiretapping is a subset of all the activities (some legally defensble and arguably necessary, some illegal) made possible by the collection of the data. A ruling on the collection of the data (which we haven’t had yet) would affect all the programs subsidiary to it, but a ruling on the subset of illegal wiretapping does not affect the larger class of activities associated with the data collection.

  12. Anonymous says:

    Thanks for that point, Libby, that’s part of what I’m trying to get at here. Some of this is legal. Some is illegal.

    Though what is new, as far as I know, is using data mining as the evidence for probable cause, which BushCo backed off doing, probably because they knew it wouldn’t fly.

  13. kim says:

    But they were able to look back into global conversations after 911 and find references to an attack, specific words referring to an attach, right, are any other anecdotal examples of this sort of thing? I’m also sure that they use frequency-spectrum type voice recognition to search conversations for particular targets.

  14. Anonymous says:


    Right, but that was probably not collected via this program. The specific story Richard Shelby leaked related to a satellite phone, which they’d have other means of collecting. And that involved someone they were already tapping–you’re talking about getting to the end of the process, which is when the NSA starts listening for words. To get from call data to those words, there’s a lot of sorting first.

    In other words, they aren’t tapping people in this program because of things they said. They’re tapping people because of suspicious patterns in their communication habits, and in the process of tapping they’re picking up individual words.

  15. Anonymous says:


    Let me make my point a different way.

    You and both agree that the warrantless wiretap program is a subset of a bunch of activity made possible by the collection of data.

    The ruling yesterday pertained to the wiretapping process. I need to go read the ruling, but from what I’ve heard, she’s talking about the legal process of tapping. I don’t think she addressed the means by which they got to that tap.

    Thus far, no one has issued a ruling on the AT&T cases, beyond whether or not they can go forward (that is, whether or not the Administration’s state secrets claim holds up). If those cases rule that that data collection was illegal (and not just a violation of written privacy policies, for example), then the warrantless wiretap program will also be illegal, because it requires the data collection. But that hasn’t happened. The only ruling thus far affects a subset of the activity.

  16. William Ockham says:


    Here’s the simplest way to make my point. The so-called data vacuuming as it is being done today is in fact equivalent to warrantless wiretapping. It may not have been when it started, but it definitely is now. While it is true that collecting metadata (e.g. phone numbers called, call length etc. for phone calls or email addresses, ip addresses, etc for internet communications) is covered by different rules than capturing communication content, if you sift through the obfuscatory comments by the administration, it is quite evident that they are not actually stripping out the content of the data and voice traffic that they are capturing. What may have started out as a pattern recognition experiment has become something else entirely. What they actually doing is capturing vast quantities of data communication contents and ignoring it until something makes them suspicious. Then, they analyze the content. In effect, they’re wiretapping everybody and just ignoring the content until some low-level analyst or a dodgy software application decides you need to be monitored.

  17. kim says:

    Although it’s all of course mostly unknown, I’ll stand by my belief that the NSA can store and scan conversations for words and particular voices from the very beginning of the process (globally and within the USA). One argument would be that actual capabilities are always well beyond public info and this is particularly true for digital technology – and ignoring mundane things like the Baltimore power grid going down.

  18. Anonymous says:


    But even if they’ve collected and saved all this data (most claims to that effect talk of it as a different program), it doesn’t mean that it invalidates the legal justification they’ve got for collecting some data. And the lawsuits against the telecoms are about the collection. My guess is that for any program like you describe the state secrets claim would be upheld, because BushCo hasn’t admitted to it.

    Even assuming what you’re saying is the the warrantless wiretap program (I disagree; I think it may exist but is considered a different program), that’s not what the AT&T programs are charging, and that doesn’t invalidate the legitimate reasons they have for being on the circuits. There still is a prior technical feasibility that is legal in some cases and has not been judged in any case (and therefore Harman is wrong).


    I think WO agrees with you. I’m not disputing the technical ability, nor am I disputing that they do it. From all the credible reports I’ve seen, that doesn’t seem to be what they did with the program ruled on yesterday.

    Of course, I’m happy to be proven wrong if you guys have quotes!!

  19. William Ockham says:


    Let’s consider internet traffic first. Based on the documents that Wired published from the AT&T suit, we know that the NSA is pulling down all the internet traffic that flows through AT&T. Here’s what the vendor of the equipment that they’re using (NARUS) claims to be able to do:

    Capabilities include playback of streaming media (for example, VoIP), rendering of Web pages, examination of e-mails and the ability to analyze the payload/attachments of e-mail or file transfer protocols.

    Proprietary directed analysis monitoring and surveillance module offering seamless integration with the NSS or other DDoS, intrusion or anomaly detection systems, securely providing analysts with real-time, surgical targeting of suspect information (from flow to application to full packets).

    None of that stuff is possible unless you’re grabbing the all the data that flows through the network. In the case of the internet traffic (including VoIP), I have no doubt that they’re getting and analyzing the headers and the content.

    I make my argument for traditional voice calls later.

  20. Kagro X says:


    The problem with the data mining part of the equation is that so long as the state secrets privilege bars justiciability, the question of its legality is moot.

    What shuts down data mining? Assuming the generally understood function of law, an order from a federal court shuts down data mining.

    What prevents the issuance of a federal court order shutting down data mining? Its non-justiciability under the state secrets doctrine.

    Nobody will infer its illegality from the parallel illegality of the TSP. Least of all this â€administration,†which continues to insist that Hamdan doesn’t touch torture or the â€inherent powers†doctrine.

  21. Anonymous says:

    But WO

    You’re using a source unrelated to these particular cases to prove your point. My point is there is a lot going on, and there are very specific cases before judges, and they don’t allege these things. And the only things that will pass the state secret claim is stuff that has been admitted by BushCo.

    Right now the AT&T case is just about collection. And the TSP cases are just about the illegal wiretapping. There’s a whole lot more activity between the two, some related, some not. But until you can connect all the dots between them (as as Kagro points out, there will remain one giant hole at the data mining location), then you can’t yet shut them all down unless you start with teh collection. But that’s teh area where they may be able to argue justificaiton.

  22. William Ockham says:

    Kagro X,

    Patience. I’m getting there. This is complicated. I don’t disagree with your judgment of the this â€administration†and its lack of respect for the law.

    Next piece of information: The call detail information. From a NYT article in May 2006:

    One senior government official, who was granted anonymity to speak publicly about the classified program, confirmed that the N.S.A. had access to records of most telephone calls in the United States. But the official said the call records were used for the limited purpose of tracing regular contacts of â€known bad guys.â€

    â€To perform such traces,†the official said, â€you’d have to have all the calls or most of them. But you wouldn’t be interested in the vast majority of them.â€

    That’s not data mining. That’s collecting a bunch of information in advance so that when you identify a suspect, you don’t have to go get at warrant to get his call detail records. The â€data mining†is being applied to the internet traffic (where they have all the details and the data is in a digital format more amenable to software analysis).

    Remember the stories about the FBI being given a bunch of leads to pizza places, etc.? The call detail database is where that crap came from. So, the question gets back to this: How do those shift supervisors that Hayden talked about decide which calls to listen in on? I think the easiest way to answer that question is to look it how the NSA has operated in the past with its surveillance of foreign communications. We know that what they like to do grab all the communications traffic they can and then pick out the stuff they find interesting.

    Here’s what I think the program looks like. The NSA has tapped into to all voice communications going into and out of the US. They filter through it based on a database of phone numbers they’ve built (through the internet program, through captured cell phones, human intelligence, whatever). When they get a hit (i.e. one of the numbers in their database) makes or receives an overseas call, they record the call. If it is an call to or from a number that’s not already in their database, it gets added to their database. If the number is a domestic number, they’ll run it through the call detail database, looking for other numbers to add. They know they can’t ever use any of this stuff to get a warrant, even from the FISC. If the court pulls the plug on the warrantless eavesdropping, the whole program is essentially worthless. It just won’t be cost-effective.

  23. Anonymous says:


    I’m confused. That NYT quote says they’re collecting all the call records. Not recording the content of all the calls. What am I missing?

    I agree with you that they are keeping that call data, no doubt about it (and in the case of internet communication, it by its very nature includes a lot more data). And they are using that data base, at the very least, to develop networks surrounding known or suspected terrorist links.

    I suspect it goes further, too. For every â€terrorist†they identity, they can pull that person’s â€profile†from the database, and get a profile of communication types. And then they can also pull similar profiles. And they’ll try tapping that. That’s where the real problem with probable cause comes in (arguably, they could do what you’re talking about on a 3-day no-warrant tap, to see if it’s necessary, because they could show the link with a real suspect, but you couldn’t show the same for a guy who just happened to have the same call patterns).

    But everything we’ve heard about the program says that once they identify someone (the 12 pizza places Mohammed Atta frequented, for example), then they do a wiretap, collecting the content of the conversations. That’s the only explanation for the descriptions of the numbers we’ve heard, that people get added to the tap list in big bunches (which by itself suggests there is a secondary step of tapping them).

    That says there are three 3 steps (at least): collection, target choice (through whatever means), and tapping. The first step is illegal (possibly) for different reasons. The third step is undeniably illegal based on FISA. And the court has said it can’t look at the second stage because state secrets applies–the court has said BushCo haven’t told us what there doing in step 2.

  24. Kagro X says:

    How long does it take to run a domestic number through the call detail database, looking for other numbers to add?

    Less than 72 hours?

    If so, then who cares if you lose on your warrant request before the FISC?

  25. William Ockham says:


    Sorry, if that wasn’t clear. I didn’t properly distinguish between the purely domestic stuff (internet communications and call detail data) and international calls. I’m giving the Administration and the NSA the benefit of the doubt and assuming that they are not intentionally eavesdropping on entirely domestic voice calls (except those VoIP calls they’re snarfing up).

    Let me put this a different way. The key to understanding this program is answering this question: Why did they need to bypass the FISC? Forget all the hoohah about agility and speed and paperwork and all the other crap they’ve thrown up as a smokescreen. The reason they need warrantless wiretaps is because not even the FISC will give them warrants based on the way they identified the suspects. If they can’t wiretap these people, the rest of it is all rather pointless.

    Let me describe a scenario. Suppose one of the guys I work with is from Pakistan. Further, suppose that one of his cousins has been identified as a terrorist (because he stole somebody’s girlfriend or some other equally lame reason). My Pakistani buddy gets a call from this cousin (hitting him up for money, it happens to Pakistani IT guys all the time). Now, my buddy here in the U.S. is on the NSA list. The search through that call detail database of his domestic calls and find that he call me at odd hours (usually because his code has hosed up on the server and I’ve got to go fix it). So, they add my number to the â€watch listâ€. The next time I make an international call, a supervisor at the NSA decides to listen. If I’m talking to one of my buddies in the U.K. about the movie I’m going to see tomorrow night (Snakes on a Plane), I imagine it’ll hit the papers as the next big terrorist plot. Here’s the rub, from the Administration’s point of view. They can’t ever tell a court about the call detail database. It’s clearly illegal and they know it. Their only hope is to keep in under wraps and pull the â€state secrets†claim every time it comes up. But, in and of itself, it’s pretty useless. They’d never get listen in on my inane calls without it (unless they can dig up my past association with CISPES, but that’s another story). If they have to get warrants for the wiretaps, the stinking edifice collapses.

  26. Kagro X says:

    But they did tell the court about the detail database. And then they told them it was covered by the state secrets privilege, and Judge Taylor agreed. So there it is, illegal or not, but either way untouchable.

    And wiretaps are for people who intend to take you to court. What are the chances your buddy’s cousin sees the inside of a courtroom? Or indeed the outside of anything? Ever again?

    Even without that little diversion, the claim is that the legality of the database is nonjusticiable. So long as that cover remains intact, they can do what they like. And I’m not even entirely sure that evidence that an illegal but nonjusticiable program uncovers couldn’t provide probable cause for a FISC warrant. What if a defendant attacked the warrant as without foundation, since the program that produced the probable cause is illegal, but no judge was willing to pass on the program’s legality because they all felt bound to recognize the state secrets privilege?

  27. Anonymous says:


    I think you and I are in total agreement. But I guess what I’m trying to get at is the difference between the cases representing people who are likely targets of this program (the CCR is representing employees who have in turn represented Gitmo detainees and similar things; the ACLU cases is representing a bunch of people who have reason to call Muslim countries), and a class action suit (where the only apparent criteria is that people use the telecom service and maybe call overseas).

    The former really gets to the heart of the illegal wiretapping problem–these are people who (like you in your scenario) are likely targets of the program. But the other people are really going after AT&T because of what happened to their calls. I’ve read the complaint and it DOES rely heavily on the warrantless wiretapping. But I suspect they did so to get somewhere–as I’ve said, the collection of data is potentially legal. (And FWIW, unless BushCo fucks up, I suspect the collection part of the suits will be dismissed, because BushCo has never admitted to that part of the program, even though it’s clear that it exist). Nevertheless, the bulk of the AT&T case talks about the technical means of collecting the data (as it would, IMO, to reach class action status). The ACLU case doesn’t even mention it. They’re not complaining that their call data has been collected, they’re complaining that, because it’s likely they’ll be tapped, their speech has been chilled.

  28. Anonymous says:

    EW – re ’falafel searches, here’s Risen:

    That is essentially what Echelon is – Echelon is the public name for what the NSA has been doing overseas for many years. They have been doing keyword searches all over the world, outside of the US in the past – using their vast computing power and their ability to monitor foreign communications. Now it’s possible they are doing the same thing inside the US. (link)

  29. Anonymous says:

    Former NSA director Admiral Bobby Ray Inman was asked â€Is this all about who-called-who?â€

    He replied:

    â€No, it isn’t. For voice communications, which are tough to search, that might be the case, he added. But with e-mail? No way.†(link).

    That’s an odd answer. He appears to be saying that they are collecting ALL content, but perhaps haven’t got the technology to search across all of the stored voice content – although it implies that they can go back and listen to specific calls.

  30. William Ockham says:


    Here’s where I think we disagree. I don’t think there is any way that the collection of the call detail records by the government is legal. Even as amended by the odious â€Patriot†act, the government has to specify a particular individual when requesting call detail records from carriers. Qwest is entirely correct in refusing to play along. As an aside, it is possible that the class action suit can breach the â€state secrets†claim by using the public statements of the (former) Qwest officials. In any event, I wish there were more judges who understood technology, because there is absolutely nothing in this program that needs to be protected for technological reasons.

    Furthermore, I think by buying into the Administration’s frame that there are multiple programs, we prolong the ability of the government to claim â€state secretsâ€. If you take a look back at the evolution of their public statements, you can see that the â€multiple programs†lie was ginned up when they realized that the AUMF rationale wouldn’t work and, that by letting Bush brag about his authoritarian toy, they had screwed themselves on the â€state secrets†defense.

    Here’s why I think the recent decision is worth uncorking your champagne (or in my case, a nice bottle of Fuller’s ESB). Three critical things were decided in our (the people’s) favor. First, she ruled that somebody had standing without having to prove the more or less impossible (that they were definitely spied on). Second, she ruled that by talking about the program publicly the government had given up its â€state secretsâ€. Third, she ruled that the program was unconstitutional, meaning that there is no way for this hapless Congress to â€fix†the problem.

    I certainly realize that any one of these issues could get reversed on appeal, but we should celebrate our victories when we get the chance. This battle with the authoritarian impulse will never end. If you wait for the final victory, you’ll never get to celebrate.