The Biggest Math Organization in the World Has a Simple Arithmetic Problem

In this post, I’m going to examine a claim made in the May 3, 2012 audit report of NSA violations. Through the magic of simple arithmetic, I’m going to show that the report misleads readers about why the number of incidents rose in the first quarter of 2012, wrongly suggesting it was an unpreventable seasonal problem, rather than pointing to the human error and fault that really explained the increase.

On page two, the report shows how many Signals Intelligence Directorate-reported incidents there are across both kinds of authorities: EO 12333 (strictly foreign) and FISA (involving US persons).

Screen shot 2013-08-16 at 10.30.37 AM

As the report acknowledges, there was an 11% increase in incidents for both kinds of authority.

But don’t worry, the report says, the increase is due to Chinese New Year, sort of.

The increase in incidents reported for 1QCY12 was due to an increase in the number of reported Global System for Mobile Communications (GSM) roamer1 incidents, which may be attributed to an increase in Chinese travel to visit friends and family for the Chinese Lunar New Year holiday.

1Roaming incidents occur when a selector associated with a valid foreign target becomes active in the U.S.

On the following page, a section provides further explanation on the roamer problem.

The largest number of incidents in the System Limitations category account for roamers where there was no previous indications of the planned travel. These incidents are largely unpreventable. Consistent discovery through the Visitor Location Register (VLR) occurs every quarter and provides analysts with timely information to place selectors into candidate status or detask. Analysis identified that these incidents could be reduced if analysts removed/detasked selectors more quickly upon learning that the status of the selector had changed and more regularly monitored target activity. This analysis indicates that continued research on ways to exploit new technologies and researching the various aspects of personal communications systems to include GSM, are an important step for NSA analysts to track the travel of valid foreign targets.

On page 6, we get a more comprehensible explanation.

Roamers: Roaming incidents occur when valid foreign target selector(s) are active in the U.S. Roamer incidents continue to constitute the larges category of collection incidents across E.O. 12333 and FAA authorities. Roamer incidents are largely unpreventable, even with good target awareness and traffic review, since target travel activities are often unannounced and not easily predicted.

In other words, the roamer problem stems from the fact that when valid foreign targets travel to the US with their GSM phones, analysts don’t know that and therefore don’t act accordingly. I think (though am not positive) the presence of the target in the US would shift a 12333 intercept into a FISA one (we’d be tracking calls to foreigners with one end in the US), and a FISA Amendments Act target into an illegal one (we’d be tracking calls with both ends in the US, one potentially involving a US person). Since this involves primarily valid foreign targets, it is not the most urgent problem identified in the report.

And, the NSA claims, it is largely unavoidable, so readers of this report should expect the relatively large numbers of roamer problems to continue.

Up to this point — far beyond where most readers will be paying attention, I’d imagine — we might believe (because the report said so explicitly) that the 11% increase in incidents stems from a problem involving valid foreign targets and reflecting an unavoidable technical problem.

It’s only when you get to page 5 and 6 that this narrative falls apart. Here’s how many roamer incidents occurred under EO 12333 for the four quarters reported.

Screen shot 2013-08-16 at 10.52.25 AM


And here’s how many roamer incidents occurred under FISA for the four quarters presented.

Screen shot 2013-08-16 at 10.53.06 AM

Adding the roamer incidents for each kind of authority together, we discover the total roaming incidents, across both authorities, look like this in the last quarter of 2011 and first quarter of 2012:

4QCY11: 582 + 87 = 669

1QCY12: 491 + 95 = 586

In fact, the roaming problem doesn’t explain the 11% overall increase in incidents at all, because the number of roaming incidents under EO12333 actually went down 19%, meaning roaming incidents across the two authorities went down 14%.

The roamer explanation doesn’t even explain the entire increase in FISA incidents, as FISA roamer incidents only went up 9%, as the report admits on page 6. On that page, the report admits that another big source of increased incidents, for both kinds of authorities, comes from database queries.

During 1QCY12, NSAW SID reported an increased of 9% of roamer incidents under all FISA Authorities. There was also a 260% increase in database query FISA Authority incidents during 1QCY12. Human Error accounted for the majority of all FISA Authorities database query incidents (74%).


Database Queries: During 1QCY12, NSAW SID reported a total of 115 database query incidents across all Authorities, representing a 53% increase from 4QCY11. E.O. 12333 Authority database query incidents accounted for 84% (97) of the total, and all FISA Authorities database query incidents accounted for 16% (18).

The report goes on to describe the root causes for these database query incidents.

Broad syntax (i.e., no or insufficient limiters/defeats/parameters)

Typographical error

Boolean operator error

Query technique understood but not applied

Not familiar enough with the tool used for query

Incorrect option selected in tool

Lack of due diligence (failure to follow standard operating procedure)

Training and guidance

Resources (Inaccurate or insufficient research information)

Thus, the front page story on this report should not be “oh, it’s all the fault of the Chinese New Year,” but instead, “oh, a bunch of human errors and due diligence (which implies fault) and research problems (which might be somebody else’s fault) resulted in the better part of the reported incident increase.”

There’s one more thing that accounts for the increase: International Transit Switch Collection, which is the problem behind the FISA Court’s October 3, 2011 finding that the program violated the Fourth Amendment (see this report, which I’ll return to). Transit Program violations went from 7 in 4QCY11 to 27 in 1QCY12. But don’t worry about that, the report says; it just reflects a different counting method.

International Transit switches, FAIRVIEW (US-990), STORMBREW (US-983), ORANGEBLOSSOM (US-3251), and SILVERZEPHYR (US-3273) are Special Source Operations (SSO) programs authorized to collect cable transit traffic passing through U.S. gateways with both ends of the communication being foreign. When collection occurs with one or both communicants insider the U.S., this constitutes inadvertent collection. From 4QCY11 to 1QCY12, there was an increase of transit program incidents submitted from 7 to 27, due to the change in our methodology for reporting and counting these types of incidents.

I’m guessing this changed methodology arose in response to the FISC opinion and also guessing that there should have been far more than 7 or even 27 violations reported before the FISC declared all this illegal.

But that still means that another important contributor to the rise in incidents between the two quarters had to do with actually counting the number of times US person communications were collected off of purportedly international transit switches.

Now, I might assume the simple math problem the NSA exhibited here was innocent.

Except it’s the NSA, the biggest math organization in the world. If I — someone who hasn’t played around with math in 27 years — can find the NSA’s simple arithmetic problem, I’m going to assume it is not an accident at all, but stems instead from an effort to hide the more serious problems deeper in the report.

The NSA: failing simple math.

Update: I should add that one other source of the increase — larger in absolute terms than the roamer problem — comes from analysts failing to turn off some or all of a wiretap of a US person when they’re supposed to.

7 replies
  1. Ben Franklin says:

    Hmmm. I wonder how Gellman beat Greenwald to the punch. After Obama’s speech last Fri and the FISA court judge acting the fool (also in WaPo) it seems Staff is rushing ahead of Bezos’ takeover.

    No doubt he would see the need for ‘prior restraint’.

  2. greengiant says:

    Call me tin hat, but whenever I travel, Rachel from credit card services calls up to get a voice check.

  3. Mindrayge says:

    Whoa, wait a minute. Executive Order 12333 is NOT limited to foreign only.

    Section 2.3 begins with:

    “2.3Collection of Information. Agencies within the Intelligence Community are authorized to collect, retain or disseminate information concerning United States persons only in accordance with procedures established by the head of the agency concerned and approved by the Attorney General, consistent with the authorities provided by Part 1 of this Order. Those procedures shall permit collection, retention and dissemination of the following types of information:”

    Section 2.4 begins with:

    “2.4Collection Techniques. Agencies within the Intelligence Community shall use the least intrusive collection techniques feasible within the United States or directed against United States persons abroad. Agencies are not authorized to use such techniques as electronic surveillance, unconsented physical search, mail surveillance, physical surveillance, or monitoring devices unless they are in accordance with procedures established by the head of the agency concerned and approved by the Attorney General. Such procedures shall protect constitutional and other legal rights and limit use of such information to lawful governmental purposes. These procedures shall not authorize:”

    Section 2.5 begins with this:

    “2.5Attorney General Approval. The Attorney General hereby is delegated the power to approve the use for intelligence purposes, within the United States or against a United States person abroad, of any technique for which a warrant would be required if undertaken for law enforcement purposes, provided that such techniques shall not be undertaken unless the Attorney General has determined in each case that there is probable cause to believe that the technique is directed against a foreign power or an agent of a foreign power. Electronic surveillance, as defined in the Foreign Intelligence Surveillance Act of 1978, shall be conducted in accordance with that Act, as well as this Order.”

    Executive Order 12333 is NOT limited to foreign only. It is solely related to authorities to collect and disseminate intelligence information and cooperation across the IC. Please don’t leave your readers with the impression that Executive Order 12333 is strictly foreign when it is not.

    The problems reported under the FISA authorities are those in which the government actually invoked the FISA authorities, likely in collection orders. These are a subset of the 12333 authorities. More likely, these are violations that are required to be reported to the FISC under those orders. The rest of the problems are under such authorities like the PATRIOT Act and other statutes that do not fall under FISA and are required to be reported under those statutes.

  4. Mindrayge says:

    I am not sure that the FISA court opinion has to do with the International Transit Switches. The term DNI is related to Internet data collections both content and metadata which goes into PINWHALE (content) and MARINA (metadata) and you can see problems with queries against both. Note also that these query problems occurred after the FISC Court decision. Based on what is in the reported issue per the FISC it seems quite likely that the problem occurred in UPSTREAM. It does seem important to me that the reference was to “upstream” and not the code name UPSTREAM as depicted in previously released slides.

    Multiple Communications Transactions likely involves all the different kinds of communications by a person across different communications types. That is, chat, text message, voice, e-mail, etc. It is not possible to unify these under “contact chaining” for example without knowing some identity information such as subscriber name or physical address. There is no way for them to tie a land line phone number to an e-mail account based on the metadata that the government claims to collect for just one example.

    One other important thing to note is that with the International Transit Switches problem it notes that FAIRVIEW, STORMBREW, ORANGEBLOSSOM, and SILVERZEPHYR are SSO Programs while in the 30 page White Paper released last Friday we have this:

    “Under all FISA and FAA programs, the government compels one or more providers to assist NSA with the collection of information responsive to the foreign intelligence need. The government employs covernames to describe its collection by source. Some that have been revealed in the press recently include FAIRVIEW, BLARNEY, OAKSTAR, and LITHIUM. While some have tried to characterize the involvement of such providers as separate programs, that is not accurate. The role of providers compelled to provide assistance by the FISC is identified separately by the Government as a specific facet of the lawful collection activity.”

    That is, last Friday the government claimed these were code names for telecommunications providers rather than programs. FAIRVIEW had been previously described by the Washington Post as being involved with the collection of mobile phone activity, particularly text messages.

    As an aside DNR refers to telephone collections both content and metadata which goes into NUCLEON (content) and MAINWAY (metadata). From the Boundless Informant FAQ we know that TUSKATTIRE is what is called the “ingest processor” for DNR collections and FALLOUT is the “ingest processor” for DNI collections.

  5. emptywheel says:

    @Mindrayge: Yup, I know that. And PAA, at least (and therefore presumably FAA), is primarily authorized by 12333 per the Yahoo decision.

    I was using shorthand, but as you suggest I think NSA thinks of the FISA stuff as that which there are limits on.

  6. emptywheel says:

    @Mindrayge: I wouldn’t get too tied to the word “telecom” provider. This is going to include a range of “providers,” and in the same way that “facility” can mean one person’s phone or an entire switch, I think this can too.

    But I agree the boundary between FISC and 12333 is vague, and probably intentionally so (bc FISC leads to minimization).

  7. earlofhuntingdon says:

    It would be a pretty inept system that didn’t match a mobile phone’s (international/foreign) number with its current (US or foreign) location, through which it accesses a telecoms provider in order to process a call.

Comments are closed.