If NSA Commits Database Query Violations, But Nobody Audits Them, Do They Really Happen?

Barton Gellman, at the beginning of the worthwhile video above, addresses something I addressed here: the only way the government can claim they haven’t “abused” the rules governing NSA activities is by treating all abuse done in the name of the mission as a mistake.

The President, like a lot of people who work for him, has a very narrow definition of two key words in that passage. One is “abuse” and the other is “inappropriately.” As the government depicts it — and this is language it’s using that it does not, frankly, explain.

Abuse — the only kind of abuse that exists would be if, say, an NSA employee were to stalk his ex-wife or spy on movie stars or something of that nature. If they are performing the mission that the NSA wants them to perform, and nevertheless overstep their legal authority, make unauthorized interceptions or searches or retentions or sharing of secret information, that is not abuse, that’s a mistake.

That’s how they get to pretend the 9% to 20% of violations in which a person does not follow the rules seemingly intentionally (these are distinct from human error and training violations) does not constitute an abuse.

With that in mind, I wanted to look more closely at what the audit report says about how errors are found, as shown primarily in this figure:

Screen shot 2013-08-20 at 10.21.25 AM

That looks pretty good on the face, with 64% of all violations found via automated alert, plus a few more — data flow analysis and traffic scanning — that involve technological review.

But this detail on the roamer problem (in which valid foreign targets continue to be targeted when they travel to the US) explains what that’s not all that impressive. 

The biggest class of technological mistakes comes from that. They actually for some reason only get quarterly reports of where a GSM mobile phone is in the world. And so you’re monitoring a foreigner using a telephone in China, the person flies to San Francisco, and you’re still monitoring. That’s against the law under FISA without a specific warrant . But they only find out every three months, and they find out sooner once someone says here I am down at the waterfront in San Francisco and they say oops.

Between FISA authority and 12333 authority surveillance, there were 586 roamer violations found in the quarter in question. If roamer violations are found — as Gellman suggests — with the quarterly report (that is, an automated alert) it would mean many of the automated alerts involve roamers. Indeed, the sheer numerical scope of the automated alerts says they are primarily roamer violations. (Adding together the two roamer-specific categories here with the automated alert accounts for 571 variations, though some of the automated alerts may be for other violations.)

Compare that to a potentially far more sensitive violation, the database query violation. That section of the report says that 70 of the 115 database query violations were found via audit. So 61% of database violations were found only through the review of another person; 84% of the violations found by auditors were database query violations.

As Gellman notes in this piece, they could automate more of the checks on database queries, but they don’t.

What this means is that (like IRS violations) short of automating this review, the government is only going to find as many database violations as they provide manpower to check.

Which may explain this detail, from Gellman’s original report on NSA’s violations.

Despite the quadrupling of the NSA’s oversight staff after a series of significant violations in 2009, the rate of infractions increased throughout 2011 and early 2012. An NSA spokesman declined to disclose whether the trend has continued since last year.

It turns out when the NSA assigned more staff to review this (though part of this staff increase must just be the staffing of the Director of Compliance position, which was instituted in 2009), they found more violations. And probably, given the role of audits in finding database query errors, more database query errors.

Which raises the question of whether the government has an adequate number of auditors now to find the database query violations (and other more intrusive violations) that occur?

While it’s a different — but related — issue, it’s worth recalling the response to Ron Wyden’s request for a hard number of the US persons whose communications are searched after having been incidentally collected. The NSA Inspector General claimed (and the Intelligence Committee Inspector General endorsed that claim) that resource limitations prevented the NSA IG from identifying how many Americans’ records get searched through this back door.

I defer to his conclusion that obtaining such an estimate was beyond the capacity of his office and dedicating sufficient additional resources would likely impede the NSA’s mission.

This response came just months after the IC IG said he had all the resources he needed to conduct his work. Since that time, we’ve learned he has been focusing on investigating leaks, not investigating whether the NSA commits database query violations.

Now, none of what the IC IG is doing says much about how the NSA IG spends his time.

But the apparent importance of audits to finding database query violations does raise the question: how many auditors does NSA have? What are their priorities? Do they have the appropriate number of auditors?

We don’t know one way or another — it’s one of those transparency™ things. But it’s a question that must be answered before anyone can assess the claims about the level of abuses violations mistakes.

16 replies
  1. Snoopdido says:

    I’ve never been really impressed by conclusory statements by auditors and their auditing that they’ve found all of the problems.

    It seems a belief system rather than factual knowledge.

    The truism is that auditors and auditing (whether done by machines or real people) can’t factually know that they’ve found ALL problems. Or to put it another way, what about the problems you didn’t find?

  2. What Constitution says:

    Not being all that technologically savvy, but it certainly strikes me that having the system set up to report only quarterly on instances of “roaming” inside the US — where US-basing is a trigger for warrant requirements — is not a flaw or a glitch, it’s a design feature which (I’m shocked, shocked to learn) serves to allow up to three months of tracking inside the US, with an opportunity to dump and excuse the tracking if it doesn’t pan out as desired. Maybe. Could that be a viable interpretation?

  3. bsbafflesbrains says:

    I believe we have crossed into the “some animals are MORE equal than others” stage. The farm is doomed.

  4. Snoopdido says:

    @Snoopdido: I guess I shouldn’t be surprised when folks like Ben Witte over at Lawfare (http://www.lawfareblog.com/2013/08/the-nsa-the-washington-post-and-the-administration/) and Jennifer Rubin over at the Washington Post (http://www.washingtonpost.com/blogs/right-turn/wp/2013/08/18/nsa-scandal-or-near-perfection/) somehow treat a NSA audit report as sacrosanct and incontrovertible evidence from on high that the NSA can do no wrong.

    It’s as if these people suspend both their memory and their judgment when it comes to auditing and auditors.

    It doesn’t seem to me that they’ve ever considered the quality of the auditing at the NSA. That it might be on the order of Bernie Madoff quality auditing or Enron quality auditing.

    And I defy anyone to convince me that with the NSA’s reported 20 million inquiries each month, that any auditor would have audited even a miniscule fraction of a fraction of those queries.

  5. earlofhuntingdon says:

    If Mr. Obama can recategorize all people within a drone missile’s blast radius as legitimate targets, he can pretend not to be abusing his powers and the law by not counting “errors” or by claiming they were inadvertent rather than part of a planned, intentional system to gather everything out there.

  6. peasantparty says:

    “As Gellman notes in this piece, they could automate more of the checks on database queries, but they don’t.”


    So, the GPS databases have to be checked, the phone call database has to be checked, the computer email has to be checked, the instant message and texting has to be checked, and EXACTLY HOW MANY DATABASES are left unmentioned? Plus, there appears to be foreign and domestic databases, and SHEESH!!!!

  7. peasantparty says:

    @Snoopdido: Nobody is watching the watchers and if those two want to pretend so then wait till their info gets leaked. The story will change quickly.

  8. Snoopdido says:

    @Snoopdido: Another one who flunks the gullibility test – David Gewirtz’s “Data-driven analysis debunks claims that NSA is out of control” – http://www.zdnet.com/data-driven-analysis-debunks-claims-that-nsa-is-out-of-control-special-report-7000019522/

    “Summary: If these numbers were reported in a corporate situation, they would be considered an absolute triumph of big data management and implementation.”

    He never once considers that maybe, just maybe, these NSA audit figures have all the flexibility and accuracy of a Bernie Madoff quality audit or Enron quality audit.

    Some people will still buy a pig in a poke even after being told it’s a pig in a poke.

  9. Valley Girl says:

    OT, except that Marcy referenced this post (this one), and this nbc article in recent tweets


    ~US doesn’t know what Snowden took, sources say~

    ~One U.S. intelligence official said government officials “are overwhelmed” trying to account for what Snowden took. Another said that the NSA has a poor audit capability, which is frustrating efforts to complete a damage assessment.~

    so, this would help explain the detention of Miranda, seizure of all devices, etc.?

  10. Orestes Ippeau says:

    Just by way of comparison, bmaz could maybe post something on the more ‘normalized’ context of specifically-targeted orders in criminal investigations. Just one example out of many I know about personally: a large metropolitan filing system apparently lost — maybe never actually filed, who knows — a wiretap order that aimed to ‘continue’ a string of orders that traced back years. Nobody noticed: the process just kept on going, authorization after authorization, each building on the product from its predecessors. Then: a whole bunch of arrests, a ‘sweep’. Most pled out early, with each, investigators, prosecutors and defending attorneys and judges all characterized them as ‘small fry’, while the higher-ups with too much at stake lawyered up. In a lengthy pre-trial motion, the unfiled authorization finally came up, and the judge through out everything that came from it and every single subsequent authorization that in any way could be said to rely on it: tens of thousands of intercepted phone calls.

    This was no anomaly. The Wire occurs in every big city and town, and there are ALWAYS such ‘errors’. But the point is: those tens of thousands of intercepted phone calls that were thrown out were just what the investigators were prepared to report into the system.

  11. orionATL says:

    recently i ran across a short video featuring john delong.

    in it he talks explicitly about how nsa is having not only to train personnel to evaluate errors but, as he put it, we are having to train our computers also. the implication was that this error catching job was not yet a smoothly running function at nsa and relied on immature computer software.

    my guess is that, to the extent nsa cares, they have come to realize that they will have to rely on software to keep abreast of error rates. otherwise, it would take the populations of india and china to do the requisite audits with humans.

  12. P J Evans says:

    If they weren’t trying to collect everything, they wouldn’t have such a big problem.
    It’s hard enough to manage a system of small databases.

  13. orionATL says:

    @P J Evans:




    we do not need this weak, unstable lego monster.

    there is no need for it in terms of solving crime or of “national security”.

    it has been developed and is being enlarged and ferociously protected, solely for the purpose of making large amounts of public money dissappear into private pockets.

Comments are closed.