Yes, Verizon, We Can Hear You Now

Screen shot 2014-01-20 at 3.20.11 AMApparently, James Clapper does not believe the information in the screenshot to the right to be classified. The name, Verizon, was left unredacted in one of the Primary Orders released last Friday (the one dated January 20, 2011). (h/t Michael)

The paragraph is boilerplate that appears, in some form, in all the Primary Orders for the phone dragnet. I had always thought the word behind the redaction was something like “the telecoms.”

Screen shot 2014-01-20 at 3.24.16 AMIt wasn’t. It appears that this Primary Order, which applies to all providers for the dragnet, applies only to Verizon.

That appears to suggest that, at least in January 2011, Verizon was the only dragnet provider.

(See below for an updated explanation: they just broke out Verizon into it’s own paragraph to limit any collection from their foreign metadata. I assume the earlier paragraph applies to the other providers.)

Now, I’m not sure what this means (I’ve got some theories, but I’m still mulling them), but it may explain why NSA Review Group member Geoffrey Stone has claimed the government get substantially less than 75% of all US traffic, but DOJ keeps telling courts that they get the whole haystack of phone records in the US. Verizon’s traffic, by itself, doesn’t constitute 75% of US traffic. But its circuits would have access to far more than just Verizon traffic. (A whistleblower has described  a wide-open Verizon circuit at Quantico.)

stormbrew-01Remember, contrary to the “Business Records” moniker, the records the NSA collects are not real billing records for much of the telecom traffic; no one has to bill for local calls for land lines, after all. So at least some of what the government obtains must be created for it. But it’s possible that Verizon strips some portion of the nation’s call metadata as it traverses its backbones.

Furthermore, if Verizon provides all this data, it explains why the providers are balking it retaining the dragnet data themselves. Not only would Verizon have to store far more than they currently do (they don’t store as much as AT&T), but it would have to fiddle with the dragnet data of other carriers, including performing the data integrity role that gives direct access to raw data.

In any case, if Verizon is still the sole provider of this dragnet data, it means it may be easier to force the end of its collection.

Update: Okay, I think I have an explanation for this now.

Up until at least March 5, 2009, all the telecoms were addressed in one paragraph starting, “the Custodian of Records.” Starting on May 29, 2009, that’s split out into two paragraphs, with the original Custodian of Records paragraph and the one we know to be specific to Verizon. We don’t have the following order, dated July 8, 2009, but we know that order shut down production from one provider because it was also producing foreign-to-foreign data; that production was restarted on September 3, 2009.

So what appears to have happened is in the End to End review, they realized that Verizon was also turning over foreign data (perhaps from Vodaphone?); this apparently was a big problem, but I’m not sure why. So they appear to have recognized they had to specify that they didn’t want (I’m guessing) Verizon’s foreign call data, at least not this way.

I assume the other paragraph names AT&T and TMobile or something like that after all.

14 replies
  1. Rexflex says:

    I have to chuckle looking at those new Verizon ads asking folks to look at a map Stevie wonder could see and asking them who has the most coverage in the US.
    Is there any way the public could find out what the telecoms bill for the service they provide to Ft. Meade?
    Kinda seems like customers should know why their shitty services are being dialed down by the government and the profits are going to a few stockholders. I believe in the Patriot act is does say compensation at the “prevailing rate”
    Marcy thanks beyond for all of your info and the translation of what it all means.

  2. Mindrayge says:

    I am not sure where you are going with this. None of the phone companies, including Verizon, are creating these call detail records for the government. These call detail records come from Signaling System 7 (SS7) which is the international standard used by telecommunications companies world wide.

    Basically what happens in SS7 is that when you dial a number the system sets up a Call Detail Record at each end of the call. Each end is considered the local switching office prior to the “last mile” to your home. This record is modified as various events within the call processing occur – dialing, ringing, no answer/go to voice mail/busy/answered, and release (hang up) of the call. These events are stored as different status values along with different data fields in the same records being populated. The records are created for every call, local or not, because the system needs them to function.

    These records are saved at each end, eventually make it to a central database, and the records are processed to create billing records later. So the most important thing to know is that billing records are created from the call detail records not the other way around. There is no situation where the phone company does not have call detail records for every call.

    So when the orders under 215 specify all call detail records they are getting information on every phone call ever made – including those where you completed dialing but hung up before there was even a ring.

    The other thing is that each individual telephone company has their own order from FISC. The orders, and the declarations are boiler-plate with only the names changing. Only a handful of companies actually have the physical switches and wires in this country. All other phone companies buy minutes and resell them so all of the call detail records for those calls exist with the companies that own the physical infrastructure.

    Cell phone calls travel over this same system, as I had pointed out previously on The Big Text Message post. Even though you are talking via radio to a tower eventually (some towers have microwave relay links to other towers) get into the national telephone infrastructre (i.e. one of the big carriers) and thus produce Call Detail Records as well.

    SS7 has a record type for handling SMS (Short Message Service) (text) messaging. When those enter the system they go to a Store-and-Forward server (no different than an e-mail server) that will attempt to get the message to its destination.

    Location information records exist in SS7 because the system needs to know what tower, and thus what end-point switch to connect an incoming call to you. These records are kept by the companies. All of them sell a service where you can monitor where a particular phone that you own is located and you can track it on a map at about 1 minute intervals. It runs like $8 a month on AT&T for example. Which is quite a racket considering that you paid for the phone and service that is generating the records they are selling back to you.

    The data listed to be provided such as communications device identifier, and truck identifier, for example, along with what would be in “etc.” are not things that end up in billing records. If all the NSA wanted was calling, number, called number, date, time, and duration they could obtain them through orders for billing records. But that isn’t what they are doing at all.

    The reason is not because the companies otherwise don’t retain this information as the government has alleged – they do, as ATT’s “HAWKEYE” database using their own custom “DAYTONA” database software is incredibly massive as it likely contains every call detail record since the system came online. The reason they do it is because billing records for any one account, which may be for multiple phone numbers, only occur once a month which means a call made on the day the billing records were generated wouldn’t be known to the NSA until some 30 days later.

    Could you imagine the hulabaloo in the court rulings that would occur if the FISC approved the government request but required the telephone companies to generate billing records daily and send those only? If the government claim is taken as the gospel truth that they only obtain the same information we see on our bills then why isn’t this one of the changes the government makes? The FCC clearly can set such rules.

  3. emptywheel says:

    @Mindrayge: Thanks for all this.

    This, though:

    The other thing is that each individual telephone company has their own order from FISC. The orders, and the declarations are boiler-plate with only the names changing.

    Is the point of this post. This is the MASTER order, not Verizon’s order. (There is the primary order, which applies to all providers, and the secondary order, which applies to specific providers.) And yet this master order lists only Verizon. That’s the point.

  4. William A. Hamilton says:

    Do you know whether former George H. W. Bush Attorney General,, and former CIA employee, William Barr, was still Vice President and General Counsel of Verizon when what you are reporting began?

  5. Mindrayge says:

    @emptywheel: It makes me wonder if any of the orders for any other provider are valid. That would be somewhat more than a minor error. Does any customer of a provider other than Verizon have recourse should an unredacted list of the parties in a primary order show up in the leaks? My understanding is that all of the telecoms have immunity.

    Do we know if this (the redaction being “Verizon”) holds up in all of the primary orders?

    Could this be a bad copy and paste job coming from the very first order and it goes to prove that no one, including the FISC judge, even bothers to read the orders before they are signed?

    I am still not sure of what the actual implications are and what that can mean in terms of attacking this via the courts.

  6. Peterr says:

    Remember, contrary to the “Business Records” moniker, the records the NSA collects are not real billing records for much of the telecom traffic; no one has to bill for local calls for land lines, after all. So at least some of what the government obtains must be created for it. But it’s possible that Verizon strips some portion of the nation’s call metadata as it traverses its backbones.

    As I read this, the question of “who owns the data?” kept coming into my head. If Verizon is stripping some portion of the data created by AT&T (for example), has Verizon stolen something from them?

    Whether Verizon did this under a court order or not, if I’m AT&T, I’m not going to be happy at all about this activity on the part of my competitor. In fact, I’m going to start to wonder if there are other things that the government is doing with Verizon, that might give them a competitive advantage somehow.

  7. orionATL says:

    so wouldn’t land lines, because no billing record for individual calls,

    be a hole in any terrorism defense effort?

    which, if so, could be taken to suggest terrorism prevention is not a high priority for nsa, if any priority at all.

  8. scribe says:

    @Rexflex: You ask:

    Is there any way the public could find out what the telecoms bill for the service they provide to Ft. Meade?

    Kinda seems like customers should know why their shitty services are being dialed down by the government and the profits are going to a few stockholders. I believe in the Patriot act is does say compensation at the “prevailing rate”

    Short answer: “No.”

    Long answer: Back during the 70s, someone fearmongered that the Soviets might possibly be able to suss out just who the government was wiretapping and how much of their traffic the NSA was capturing by means of parsing through the SEC-required annual and periodic reports and the balance sheets of the telco. Recall, the Bell System had not yet been broken up at that time, so there was only one telco really relevant to wiretap, particularly when it came to international traffic. So, in response to the fearmongering (not really rational, when you think about it, because annual reports and SEC filings can be pretty damned opaque when the companies want), there was an amendment made to the relevant laws and the telco was exempted from publicly reporting their income from government (national security) wiretapping. In short, the telecoms are running two sets of books – one for themselves showing just how much they make from helpng the government, and one for public consumption – and doing it with official sanction. By now, they might be required by law to run two sets of books and not report the wiretap income.

    As to the master order discussion of why there’s only one for Verizon only, I suspect one would have to get deep into the weeds of the breakup of the Bell System and subsequent creation and later re-mergers of the Baby Bells to find out just who controls the switches and nodes of the backbone whence the tapping comes. I can see – it seems pretty rational or even mandatory, in fact – that the initial breakup of the Bell System would have contained a provision addressing this. Since Verizon’s lead forebear was Nynex/Bell Atlantic and that Baby Bell covered the territory containing the main network control center (Bedminster/Netcong, NJ), Bell Labs (suburban NJ), main (international) traffic generators (NYC, the NJ offshore cable junctions), Fort Monmouth (a big Army Signal Corps installation on the Jersey Shore that mysteriously escapes BRAC shutdowns time after time), the big NY-DC and transcontinental mainline cables and (IIRC) the Maryland/DC area, I would suspect NYNEX/Bell Atlantic would have gotten the nod to run the backbone system.

    In breaking up what had been a unified Bell System, where there was One Way of doing things everywhere, the big fear would be that one or more of the Baby Bells would skimp on maintenance and upgrades so as to goose their balance sheets. Similarly, another fear would have been that different companies in charge of parts of the system would have inevitably led to differences in the system – different equipment, different software, different upgrades at different times – that would have only created problems for both daily civilian operation and wiretapping. The result, over time, would have been degradation of the US telecom system to something approximating Third World status. The only way to avoid network chaos would be to have had only one company responsible for setting network standards and maintaining the real backbone.

    So, I’m speculating that Verizon’s corporate forebears were assigned the main responsibility for maintaining the backbone. In Verizon/NYNEX/BellAtlantic territory, it would be through its own employees or contractors. In other Baby Bells’ territories this is probably through employees or contractors ostensibly employed by the other Baby Bells but whose instructions come from one centralized place somewhere in Verizon’s basement and whose pay ultimately comes from Verizon. Recall, the early anti-dragnet suit about the modifications to the cabinet in SF being done to hook up the splitters and such contained the detail that the employee was given instruction sheets on which wires to disconnect or connect and in which order and to/from which devices. No provenance would be needed to carry out such instructions on a work order and it’s likely none would be given so they easily could have come from Verizon. Any pay or money issues relative to the work being done in areas outside Verizon’s (Baby Bell predecessors’) geographic reach could have been dealt with by pretty simple accounting.

    But, I’m deducing Verizon’s predecessors got the job of maintaining the main backbone using either their employees or employees of other companies who they pay for and direct through cutouts and that the same obtains today. So, there would only be a need for one Master Order because only one company would be doing the work of splitting off the backbone traffic and sending it to NSA. I dunno whether that’s the case, but it seems reasonable to me.

  9. emptywheel says:

    @Mindrayge: THe point is, this seems to suggest that there may be no other providers.

    That’s the reason I went back to Stone’s comment, because for them to be taking (or for Stone to think they’re taking) substantially less than 75% of all the phone data in the US, they can’t even be taking just Verizon and AT&T (which I might have believed, as it woudl give you virtually all in any case with the second end of the call likely being one of the two), because that’s about 65% of the market right there.

    I don’t think this is a cut and paste error. As I’ll show hopefully today, there were a lot of changes in the primary order along the way, particularly in the time period in which this order came out (because they were busy cleaning it up).

    As for the other providers, if any, they just get the Secondary order, so they’re safe.

    FWIW, I partly wonder whether Verizon is doing this for free and AT&T demanded payment, which required another system (again, as I’ll lay out later).

  10. emptywheel says:

    @Peterr: That’s why Myndrage’s reference to the SS7 is important. It may mean they’re doing the equivalent of what they do with SWIFT, is access the joint record, and they just happen to get that from Verizon because Verizon is willing to hand it over. If that’s right, then it’s not AT&T’s possession, it’s common.

  11. Peterr says:

    @emptywheel: But even if they happen to get it from Verizon and not AT&T, I suspect that doesn’t make AT&T feel any better about close cooperation between the government and one of their competitors. They’ve got to be wondering if Verizon isn’t getting some help from govt engineers, and using what they learned (or were given) in other ways not related to the NSA program and that might give Verizon a technical advantage over everyone else.

    Oh, but I’m sure the govt requires that Verizon put any hypothetical technical help they’ve been given behind a firewall, away from the rest of their company, and that verbal assurances to this effect will make all the rest of the telecoms breathe a sigh of relief.


  12. lefty665 says:

    Hey geography buffs, what are the locations shown in the Stormbrew graphic and why?

    Looks like it starts in NYC, goes to Raleigh, the yellow sun is ???, the oval could be Oak Ridge, the gray circle Ft. Gordon, San Antonio data center, L.A… The map is not to scale, but the triple hit is clearly inland and seems likely to be Bluffdale.

  13. Return Call says:

    A lot of good Verizon/ATT/Spring information in Joe Felton’s testimony. Nice screenshot of Analyst’s Notebook, the most common skill cited in that massive ongoing security breach, LinkedIn resumes (google search terms like ‘Sigint DISHFIRE’):

    #3 See Siobhan Gorman et al., U.S. Collects Vast Data Trove, WALL ST. J., June 7, 2013, (“The arrangement with Verizon, AT&T and Sprint, the country’s three largest phone companies means, that every time the majority of Americans makes a call, NSA gets a record of the location, the number called, the time of the call and the length of the conversation, according to people familiar with the matter.

    AT&T has 107.3 million wireless customers and 31.2 million landline customers. Verizon has 98.9 million wireless customers and 22.2 million landline customers while Sprint has 55 million customers in total.”).

    #8 Public Safety & Law Enforcement Operations, International Business Machines (last visited Aug. 22, 2013), (“IBM® i2® solutions help law enforcers to turn huge volumes of crime data into actionable insights by delivering tools for tactical lead generation, intelligence analysis, crime analysis and predictive analysis.”);

    see also Defense and National Security Operations, International Business Machines (last visited Aug. 22, 2013), (“IBM i2 solutions for military and national security organizations have been used across the world to process and analyze the vast quantities of information that they collect, to generate actionable intelligence and to share insights that help identify, predict and prevent hostile threats.”);

    see also Pen-Link, Unique Features of Pen-Link v8 at 16 (April 17, 2008), (“Many U.S. Federal Law Enforcement and Intelligence agencies have acquired agency-wide site license contracts for the use of Pen-Link in their operations throughout the United States…Pen-Link systems are also becoming more frequently used by U.S. intelligence efforts operating in several other countries.”).

    #9 Case Studies: Edith Cowan University, IBM i2 Solutions Help University Researchers Catch a Group of Would-Be Hackers, International Business Machines (Mar. 27, 2013), (“Analyzing this volume of data is nothing new to many law enforcement users who routinely analyze tens of thousands of telephone records using IBM® i2® Analyst’s Notebook®.”).
    10 Course Description: Telephone Analysis Using i2 Analyst’s Notebook, International Business Machines (last visited Aug. 22, 2013), (“This intermediate hands-on 3- day workshop focuses on the techniques of utilizing i2 Analyst’s Notebook to conduct telephone toll analysis…Learn to import volumes of call detail records from various phone carriers, analyze those records and identify clusters and patterns in the data. Using both association and temporal charts, discover how to use different layouts and more advanced tools to analyze telephonic data quickly and effectively.”).

    #11 See Pen-Link, Unique Features of Pen-Link v8 at 4 (Apr. 17, 2008), (describing the capability to import 170 different data formats, used by phone companies to provide call detail records).

Comments are closed.