3 Tech Issues the Non-Technologist NSA Technical Committee Needs to Address

A number of people are asking why I’m so shocked that President Obama appointed no technologists for his NSA Review Committee.

Here are three issues that should be central to the Committee’s discussions that are, in significant part, technology questions. There are more. But for each of these questions, the discussion should not be whether the Intelligence Community thinks the current solution is the best or only one, but whether it is an appropriate choice given privacy implications and other concerns.

  • Whether the Intelligence Community can accomplish the goals of the Section 215 dragnet without collecting all US person metadata
  • Whether the NSA can avoid collecting Multiple Communication Transactions as part of upstream collection
  • How to oversee unaudited actions of technical personnel

There are just three really obvious issues that should be reviewed by the committee. And for all of them, it would be really useful for someone with the technical background to challenge NSA’s claims to be on the committee.

Whether the Intelligence Community can accomplish the goals of the Section 215 dragnet without collecting all US person metadata

One of the most contentious NSA practices — at least as far as most Americans go — is the collection of all US person phone metadata for the Section 215 dragnet. Yet even Keith Alexander has admitted — here in an exchange with Adam Schiff in a House Intelligence Committee hearing on June 18 — that it would be feasible to do it via other means, though perhaps not as easy.

REP. SCHIFF: General Alexander, I want to ask you — I raised this in closed session, but I’d like to raise it publicly as well — what are the prospects for changing the program such that, rather than the government acquiring the vast amounts of metadata, the telecommunications companies retain the metadata, and then only on those 300 or so occasions where it needs to be queried, you’re querying the telecommunications providers for whether they have those business records related to a reasonable, articulable suspicion of a foreign terrorist connection?

GEN. ALEXANDER: I think, jointly, the FBI and NSA are looking at the architectural framework of how we actually do this program, and what are the advantages and disadvantages of doing each one. Each case, as you know from our discussions, if you leave it at the service providers, you have a separate set of issues in terms of how you actually get the information, then how you have to go back and get that information, how you follow it down, and the legal authority for them to compel them to keep these records for a certain period of time. So what we’re doing is we’re going to look at that, come back to the director of national intelligence, the administration, and then to you all, and give you recommendations on that for both the House and the Senate. I do think that that’s something that we’ve agreed to look at and that we’ll do. It’s just going to take some time. We want to do it right. And I think, just to set expectations, the concern is speed in crisis. How do we do this? And so that’s what we need to bring back to you, and then, I think, have this discussion here and let people know where we are on it. Anything that you want to add?

REP. SCHIFF: I would — because I would strongly encourage us to vigorously investigate that potential restructuring. Even though there may be attendant inefficiencies with it, I think that the American people may be much more comfortable with the telecommunications companies retaining those business records, that metadata, than the government acquiring it, even though the government doesn’t query it except on very rare occasions.

GEN. ALEXANDER: So it may be something like that we bring back and look at. So we are going to look at that. And we have already committed to doing that, and we will do that and go through all the details of that. [my emphasis]

Some of the technical issues raised about this in other venues pertains to the different ways the telecoms store their data. In particular, how they collect VOIP data must be a particular challenge.

The IC has already lied about two issues related to this issue: First, about the history of legal regulations on data retention. And about what they really mean by “speed.” Thus, even if they weren’t already predisposed to pick the easiest solution — or the one that might have benefits down the line if a President decided to use it for domestic “terrorists” — it would be useful to have a technologist on the committee to challenge NSA’s claims about timeliness or data compatibility.

For those reasons, this makes the 215 dragnet solution the perfect issue for the NSA review committee to review. The technical issues here might be simple enough for Richard Clarke to address (and he has express concerns about the dragnet). But in any eventual challenge to NSA’s claims, it’d be really useful to have experts on data mining making the argument.

Whether the NSA can avoid collecting MCTs as part of upstream collection

When John Bates declared part of the NSA program unconstitutional on October 3, 2011, a key issue pertained to whether or not the government could avoid collecting entirely domestic communications — whether Single Communication Transactions or Multiple Communication Transactions — as part of its upstream collection.

Bates appeared particularly skeptical about the government’s claims about MCTs.

Screen shot 2013-08-28 at 3.07.50 PM

 

In his discussion (see page 58), he assumes “for purposes of this discussion” that government assertions about the technical challenge are true. But then he includes a long footnote modifying that assumption, which has been entirely redacted save his exhortation that “it is incumbent upon NSA to continue working to enhance its capabilities to limit acquisitions only to targeted communications.”

Screen shot 2013-08-28 at 3.05.33 PM

 

I don’t think Bates entirely buys the government’s assertions. Technical folks here and elsewhere have also challenged the claim that the government can’t break up MCTs and destroy the US person content.

Moreover, this is an issue that goes to the core of the government’s deceit on this issue: given that prior minimization procedures made special exception in data retention guidelines for this collection, it is not credible that the government only came to understand the problem with the collection in May 2011, when they alerted FISC. Given that misrepresentation, it behooves the committee to assess NSA’s claims on this issue skeptically.

Finally, this is perhaps the most important issue going forward, given that many in the national security establishment — including Richard Clarke — envision using this technique (though sorting for malicious code rather than foreign intelligence selectors) to combat cyberattacks. The NSA has clear incentives — to accomplish its domestic cybersecurity mission, not its foreign intelligence collection mission — to pretend it can’t help itself here. It’d be really useful to have someone who knows the real technical limitations on the Committee to rebut Clarke.

How to oversee unaudited actions of technical personnel 

Finally, there’s the problem of what to do with the 999 other Sysadmins and other technical personnel who have access to enormous amounts of sensitive data but who are not currently overseen adequately.

This problem is probably not limited to Sysadmins — according to the Primary Order on the telecom dragnet, other technical personnel massage the database of all phone-based relationships in the US before the (audited actions of) analysts get it. Indeed, these technical personnel managed to misplace 3,032 dragnet files on their own server. We know this data didn’t get destroyed in timely fashion, but since this access is not audited, we don’t know what else happened to it.

And aside from the fact that so much recent reporting talks about how embarrassing NSA’s security was, there’s another reason why outside technologists should provide input on how to fix the problem. DOD has a serial problem with not addressing the threat posed by removable media, having refused to fix the problem after malware got introduced into DOD systems via a thumb drive in 2008 and after then Bradley Manning took entire databases on a Lady Gaga CD. Sure, I’m sure SAIC and Booz would love to get paid billions to advise DOD how to fix this problem, but wouldn’t it be better to hear the advice of someone not trying to get rich off of it?

This is another are where, left to itself, DOD has historically been inclined to settle for ease of use rather than security (indeed, testimony at Manning’s trial made it clear that was precisely the thought process). But DOD should at least hear an independent voice on this front, one with some technical knowledge about how DOD might fix this persistent problem.

The need for an outside is all the more urgent given Keith Alexander’s stated preferred solution is to fire 90% of the Sysadmins (which seems to both misunderstand the problem and ignore some big problems with the proposed solution). It sounds like NSA wants to adopt a technical fix to the Sysadmin problem (they haven’t acknowledged the problem with the other technical personnel yet). So it would be far better to have a technical person to explain why that’s probably not going to work.

Marcy has been blogging full time since 2007. She’s known for her live-blogging of the Scooter Libby trial, her discovery of the number of times Khalid Sheikh Mohammed was waterboarded, and generally for her weedy analysis of document dumps.

Marcy Wheeler is an independent journalist writing about national security and civil liberties. She writes as emptywheel at her eponymous blog, publishes at outlets including the Guardian, Salon, and the Progressive, and appears frequently on television and radio. She is the author of Anatomy of Deceit, a primer on the CIA leak investigation, and liveblogged the Scooter Libby trial.

Marcy has a PhD from the University of Michigan, where she researched the “feuilleton,” a short conversational newspaper form that has proven important in times of heightened censorship. Before and after her time in academics, Marcy provided documentation consulting for corporations in the auto, tech, and energy industries. She lives with her spouse and dog in Grand Rapids, MI.

32 replies
  1. earlofhuntingdon says:

    Thanks. Nicely done.

    Having tech expertise on staff would be necessary. It would also be inadequate, if the political appointees lack the tech. expertise to use staff input to its greatest rhetorical and political and negotiating effect. That’s especially true when staff are often excluded from secret meetings, where even political appointees are not allowed to take notes or allowed sufficient time to review, analyze, question materials or presenters claims.

    These sorts of committees are about democracy theater, not about common sensically holding political power to account. One way around that is to lobby for greater expertise on such panels, greater resources (time, money, staff), and greater independence. Governments resist that strenuously, allowing only what they think the public will demand. Public, demand more.

  2. earlofhuntingdon says:

    Audits would be an essential step in making these programs effective. The current approach resembles the Vietnam era policy of “Gather everything and let god sort it out.” (That policy purported to authorize Americans’ killing everybody and then letting god or the devil sort out destinations in the afterlife; it was easier than telling friend from foe and yielded great body count statistics for military careerists.)

    Audits, however, are after the fact. What seems clear is that the so-called management systems in place here – expertise, management-to-staff ratios, staffing levels, backgrounds, protocols – are woefully inadequate.

    The government and its contractors are either incompetent or prefer to look that way in order to hide practices that put no limits on data access or use. Or it could be both. Given the scope, breadth and cost of these programs, that’s unacceptable.

  3. earlofhuntingdon says:

    Wholesale firing of sysadmins is a variation on, “It was only a few bad apples, Senator.” It would blame staff for severe management failures. It’s a widespread problem.

  4. jerryy says:

    One quick quip type of answer to Rep. Schiff is that the telecom industry does not want the expense and hassle nor is their security regarding customer’s data any better than the NSA ( see: http://www.wired.com/threatlevel/2013/03/att-hacker-gets-3-years/ ).

    By his approach, Rep. Schiff does skirt the implication that it is okay for millions and millions of innocent people’s privacy being violated, just whom does the violating is the question.

    They should back up and explain why the dragnet is not some immense affront to civil rights and liberties, that it is not some wholesale disregard for our constitutional rights. Even the faint figleaf known as the FISA court has admitted that it cannot exercise oversight for these programs and Congress does not begin to put in adversarial safeguards in the process.

    If the various authorities have reason to investigate anyone, and they convince the appropriate judges / grand juries etc. to issue warrants, then the technical means to tap into the communications of the suspects are quite easy, there is no need to add in the folks living in the next county as well.

    I am not pretending there are not bad folks out there that should be investigated, just reminding the notion that gets heard here a lot: ‘if you are looking for needle in the haystack do not add more hay’ is very valid. Also, I am mindful that while it was never really settled in the press whether or not the Russian authorities gave enough warning about the Boston Marathon bombers, the point is that all the dredging of the dragnet did not stop the actions, but apparently the LOVEINT surveillance got some results. If, instead, the warning were heeded…

    How about the staff of the EFF being included in the discussions, or heck, maybe someone could talk Richard Stallman AND Bruce Schneier into giving some input, the firestorm that would erupt would be quite sobering.

  5. person1597 says:

    The so called “trusted third parties” who service intercept configurations will no doubt find a way to lobby this committee on the merits of their proprietary control solutions, as you suggest.

    Maybe if a special master in lunacy is appointed during litigation over all this, we’ll find out more… but I doubt it. Perhaps, at least the budget will be public. Or not.

  6. orionATL says:

    this committee/commission is just a tactic by obama admin to create the impression of trustworty-ness.

    it is fine to make recommendations on technical matters that seriously need addressing, but such recommendations would be better made as criticisms of the entire specious “evaluation/investigation”.

    not having technically competent members of the commission is just fine with the politicians behind the idea; they have zero interest in having the current program critically evaluated or improved.

    the central criticisms are:

    – all members are insiders publicly favorably disposed to the program.

    – not a single individual with a strong background in both constitutional rights of citizens AND the technical background to criticize the current arrangement AND to suggest viable alternatives serves on the committee.

    the commission is fundamentally a palliative designed mostly, i would guess, to provide a report to protect congresscattle from constituent criticism.

    the core issue of the dragnet is that the current electro-mechanical spying organization is prima facie a violation of constitutional rights.

    a very important corollary issue is that there are most likely dozens of other ways to do law enforcement,but none are actively being discussed.

    a second important corollary question is: why have we not been told by the president why this spying program is needed?

    answer: it has no fundamental purpose nor does it meet a fundamental government need outside of a) gov departments appearing effective irregardless, b) uses in war and quasi-war, and c) sources of income for politicians’ re-election and personal income for contractors.

    this dragnet is purely a policing program to make the fbi’s job and doj prosecutors jobs a lot easier – that’s what it is about and that’s all it’s about.

    there is no “there” there.

  7. orionATL says:

    @orionATL:

    the most important benefit of this panel is to protect those who created it – the prez and his spying comrades.

    while the panel is “deliberating” presidential/nsa spokesmen can simply say “there is a review in progress”.

    afterwards, spokesmen can say “the report says ….”

    this panel is a backfire to control the conflagration that was developing over spying on all americans without warrant :))

    let’s see if journalists let them get away with this gambit.

    even if they don’t, there is the problem of citizens’ short attention span and impatience with a long-running news story (not involving sex or murder).

  8. Greg Bean (@GregLBean) says:

    @emptywheel, I understand why you feel a technologist is required on the committee; to push back against the lies we are hearing. But, as a technologist I am pretty confident that we are hearing lies, and will elaborate below, but will continue to argue that turning this into a technological debate about what can be done is not going to address the primary issue.

    In brief, whether the technology exists to do otherwise or not, whether it is hard or easy, is a distraction from the central issue that it is illegal. Further, that it is counterproductive, as the East Germans will attest.

    If we take our eye off that fact and debate how hard or easy it is to do we are being played for suckers.

    Now to elaborate on the technological push-back:

    1/ Is a dragnet req’d? Does the NSA need it’s own copy? Does NSA having its own copy provide speed? Does the NSA having a copy enhance data mining capabilities? The answer to all these questions is NO.

    The alternative of the telecoms holding the information can deliver the same result.

    There is no getting around that the information needs to be collected. But the telecoms holding it, and holding it in silos that are not consolidated, is a good solution. It will deliver everything the NSA needs while keeping it out of their hands until and in-case it is needed and shown to be needed by a warrant.

    How can this be done. The technology to do it is in heavy commercial use already by organizations like the NYSE. Data virtualization products from companies like Composite and Denodo overlay the actual data so that no copy is required, though the virtualized data appears as if it is a live copy. Then, Semantic Web technologies overlay this virtual copy with ontologies that give it consistent meaning. This then enables search engines with a capability to extract only the details that satisfy the specific query (as defined by the warrant), to do so at speed, and to deliver that information to data mining tools that can use it for mining as extensively as they wish.

    In short, the NSA absolutely does not need a copy.

    2/ Can MCT collection be avoided. Yup. The architecture I described above will do this. And to expand on this. Data visualization software can be used to virtualize almost any information, including VOIP, image, handwritten notes, etc and when overlayed with ontologies that give it consistent meaning, it all, any information in any format becomes ‘known’. Imagine a babelfish for electronic translation and you will get the idea.

    So, MCT’s are just like any other selectable component. They’re not inseparable, in fact, just the opposite, they’re easily separable if the ontology ‘codifies’ them as MCT’s.

    And 3/ Managing access. Well this is also doable but there are two pieces required. The first piece is to use the existing (already proven deficient) security. It is not useless, just not comprehensive. The second piece is to use a Semantic Web security ontology that will overlay the data with a consistent access and use ‘braces’. This latter component is immature but good work is being done on it at DERI, Galway by various PhD’s.

    The is however a very simple fact that cannot be avoided. No system is safe from malicious attack by trusted staff. And maintaining that trust is not done by heavy handed monitoring while the role models in the organization are widely known to be Lying Keith’s, Lying Obama’s, and Lying Drone Czars. Integrity is contagious, as is deceit.

    So, in summary, are the NSA using the old technologist’s bullshit baffles brains strategy to make the technology all appear magical and using that to argue that anything else is nigh impossible? I say yes, and I say it is bullshit.

    But then, any reasonably clued in technologist would say the same.

    Does this mean that what the NSA is doing is legal or illegal, moral or immoral? No, technology is not the issue.

    We all need to keep our eye on the ball and not let the NSA hide it behind the pokery-jiggery of technological black magic.

  9. orionATL says:

    @Greg Bean (@GregLBean):

    very nicely said.

    very useful for the specific technological alternatives described. i wonder why none of this has been discussed before – maybe too much heat and not enough light in the debate to date. of course, gov has no interest in assessing alternative ways of doing things.

  10. jerryy says:

    @Greg Bean (@GregLBean): Greg, you are not making sense.

    First you (rightly) say that the dragnet(s) are not needed but then turn around and say they are, but the telecoms should do it and handle all the details. [As I noted earlier in an above comment, the private sector is not conscientious about protecting privacy of current commercial customers.] Privatizing the destruction of privacy, etc., does not make it okay.

    Further on in your comment, you ask if what the NSA is doing is. Immoral or illegal and then decide ‘no’, based on a view of technology. What the NSA is doing IS immoral and quite likely illegal (which the judicial system will decide), HOW the NSA is doing what they do is not entirely relevant. ‘Can does not mean should’ is more applicable.

  11. orionATL says:

    @jerryy:

    i don’t know what greg bean intended, but my inexpert understanding on telecom involvement was that they receive communications as their business. they also store them – i think (for three months). having telecoms retain data for a longer time in the same “raw” form in which they were received would seem to be non-invasive.

    coupled with the demand that nsa/fbi obtain warrants for specific individuals before receiving their records, this “telecom solution” might satisfy statists and law enforcement while bringing out-of-control law enforcement search authority back into the realm of honest 4th amendment protection.

    by “honest” i mean non-sophistical, without legal flim-flammery, without public lying by officials about honoring constitutional guarantees.

  12. jerryy says:

    @orionATL: By various accounts, what is being stored depends on the telecom, what type of data is flowing through their network, which MNVO (multiple vertical network operator) is tagging along, which civilian or federal authority is submitting requests, etc. Gag orders make obfuscation easier.

    This data is not really protected. Not really.

    Consider the following situation: The insurance company underwriting Horace’s Regional Cell Phones decides the liability is more than they want to handle, so they stop covering HRCP, which causes them to have to turn to more expensive measure to operate. One thing leads to another and the local bank steps in and take over. The bank now owns all that data. Do you want them deciding how loans should be doled out based on who is swapping the best nude photos via text messages to each other?

  13. jerryy says:

    @Greg Bean (@GregLBean): A subsequent re-reading of your comment’s ending shows me I misread the second point you typed, as in you did not say what I thought, you were not saying the NSA’s actions are moral and legal. The phrasing of the next sentence hid the period you have placed in there.

  14. Greg Bean (@GregLBean) says:

    @jerryy: Jerryy, you misunderstand me and or have slightly different definitions than I do.

    To me a dragnet (think fishing) is a single massive destructive scoop gathering everything, like what the NSA is doing. The telecoms, each holding their collected information in isolated silos, is not a dragnet. Using an overlay as I describe (data virtualization, ontology, search engine, etc) to selectively consolidate the required (by warrant) information, never creates a dragnet, and provides a mechanism that controls law enforcements overreach, as OnionATL rightly points out. So, no dragnet. Controlling how corporations use and abuse private information is a whole other conversation, but is certainly worth discussion.

    As to my comment about moral/immoral, legal/illegal, I was not suggesting what the NSA is doing is not both immoral and illegal. Just the opposite, I agree with you, it is both immoral and illegal. What I was saying is, having described an alternative architecture that pushes back against the NSA’s claims that technology makes them do it the way they do, I have not somehow held them accountable for their behaviour. I have shown that they lie about that claim that they are technology driven but in no way does that force them to accept what they are doing is immoral or illegal. It just shows they lie about technological capabilities.

    Until they accept that what they are doing is unacceptable (illegal and immoral), it doesn’t matter what technology is used. Debating technology is a distraction.

    Though, I can see EW’s point, removing their ability to lie about the technology backs them into a corner where they are more likely to accept the illegality/immorality of their actions.

    So maybe EW is right, break down their lies, all of them including technological ones, one at a time, until they are left without a defense, and then maybe they will admit their guilt.

    Ahhh, it’s a tangle web they weave.

  15. jerryy says:

    @Greg Bean (@GregLBean): Gill net vs trawl net? Global vs local? Town gossip vs listening post? It may be differing definitions or distinction without differerence :^)

    AT&T is a larger telecom than Sallie’s Phones, but the discernible real distinction is only in scope of magnitude. Without the few you cannot get to the many.

    As I mentioned I did misread your ending and I apologize for that. We do agree on many points.

  16. Greg Bean (@GregLBean) says:

    @jerryy: Our postings crossed, so I only saw your post at 18 after I posted at 19.

    Your questions were useful, hopefully my clarification helped. As a technologist I do not have the writing skills that someone like EW has and often need to clarify. Thx.

  17. Greg Bean (@GregLBean) says:

    @orionATL: Orion, your comment raised a niggling issue with me. The technology I mentioned is not necessarily alternative to what the NSA is using. After reading through some of the NSA technology details I mentioned in this post http://www.emptywheel.net/2013/08/27/the-no-technologist-technology-review-panel/#comment-611104 I concluded that they are likely using much of what I’ve described. They mention ontologies, the Aladdin and Renoir products look very similar to what I’ve identified as Semantic Web technology, and other pieces, under the Advanced Computing heading, also seem to fit.

    I was going to let this point go except it highlights that if they are using the technology I described, (called by a different name) but using it in a different fashion, they must know they have other options and have rejected them. In short, they are using the technology but are saying it cannot do what I know it is capable of. And that can only be to support their lies.

    They don’t need to look at alternate technology, they just need to use the technology they are currently using differently, in the fashion I laid out.

    I can only conclude that they want to have their own copy of information and will deny their technology permits them to do otherwise so they can achieve this. Not surprising really.

    BTW, I don’t seem to be able to get to those NSA links today. :-) Also not surprising.

  18. Bill Michtom says:

    @earlofhuntingdon: While the public is demanding more, why doesn’t it (we) demand that the excuse the government uses to get our private data from telcomms–it’s not really private–be changed by passing a law that makes it private from the government?

  19. Bill Michtom says:

    @jerryy: “How about the staff of the EFF being included in the discussions, or heck, maybe someone could talk Richard Stallman AND Bruce Schneier into giving some input, the firestorm that would erupt would be quite sobering.”

    Excellent–which, unfortunately, means it will never come close to being implemented.

  20. earlofhuntingdon says:

    @Bill Michtom: Good idea. This government has, however, over the course of several administrations, fought strenuously against legislating anything remotely like the privacy regimes legislated throughout the EU, Canada, Japan, Australia, etc. Congress seems entirely beholden to its lobbyists, who do its work, write its laws and regulations, and raise re-election money in conveniently large, big box store-size packages (a form of legalized bribery that is more addicting to both parties than crack cocaine).

    The corporations funding those lobbyists (and “think” tanks and academics) are heavily invested in their “we don’t need no stinkin’ privacy” regime. They make billions off of our not having adequate privacy protections every year. Billions that customers never see but pay for, on top of the nominal charges for products and services.

    Your cable company, for example, claims ownership of your specific viewing data: what you record or watch, when, for how long, etc. (Those cable boxes and digital recording devices feature two-way reporting.) Your car, especially higher cost models, ordinarily have GPS tracking (regardless, as with phones, of whether you pay for user functionality). Car engine management modules also record detailed information. Who owns that? Good luck getting GM, for example, to admit its products record such data, let alone getting them to allow you past their encryption to see it, delete or stop it being reported. (Your insurer won’t have such a hard time; think about that next time you file an accident claim.)

    Needless to say, cell phones record and report extraordinarily detailed and private data, which is commercialized by phone companies. Their customers pay big money for it. So you end up paying a big monthly bill, and pay through the nose via lost privacy, while unknown companies use and profit from your data.

    That’s the proverbial tip of the iceberg. With big money comes big power, including political power. Consumers have neither. But they could resurrect data privacy as an issue and scream loud and hard to protect it. It’s why we have, for example, weekends and holidays, civil rights legislation, along with seat belts and safer cars.

  21. Greg Bean (@GregLBean) says:

    @Bill Michtom: Bill, it’s actually trivial once you have an example. And I will explain it simplistically below, but before I do, what is more important, is that it answers the question, “Who has time to read / assess all of the stuff the NSA is collecting”. The answer is ontologies permit computers to do the assessment.

    That is why it is so powerful and why I can guarantee NSA is using it; they do not have the personpower to manually assess the volume so unless it can be codified so that a computer can do so, the information is useless. Ontologies are the key.

    Geek alert!

    A common ontology that everyone is very familiar with is the ancestry ontology. By defining the rules (simplistically: mothers are female, fathers are male, husbands are male and have wives who are female, sons are male and have sisters who are female, etc for daughters are female and have mothers who are female, etc for nieces, nephews, uncles, aunts, and on and on, lots and lots of rules and exclusions ex. fathers cannot be mothers, etc) we provide a computer with the capability to infer/deduce how everyone is related. Note: I said EVERYONE!

    You and I would have a hell of a time figuring this out manually, but computers can do it in seconds.

    What we end up with is called a graph, not a statistical type graph but a diagram that has links (showing relationships) and nodes (showing entities). We see the term graph used throughout the NSA discussions, and they are referring to diagrams of links and nodes that show relationships.

    So, to keep this brief, create an ontology that defines the rules, overlay it on the data that fills in the values (links and nodes) and bingo, a computer can work out (infer/deduce) almost anything we want to know (given enough information ala NSA dragnet).

    Who is talking to who via nth levels of separation, who is selling what to who and via what channel, or scientifically, what genes are related to which disease and via what intermediate relationships with other genes.

    So, it is common technology, in wide use, with great commercial value, but when applied by the NSA to monitor everyone in the world, it starts to look very creepy.

    I think you probably get the picture, but am happy to elaborate if there are specifics.

  22. Greg Bean (@GregLBean) says:

    And just as a further thought, the DoD wants to use the capabilities I’ve described above to allow them to look across 400 HR databases and select the single candidate with the best skills for any task. (according to Dennis Wisnosky http://en.wikipedia.org/wiki/Dennis_E._Wisnosky)

    That’s right, 400 separate HR databases, all within DoD.

    So, if DoD can pull data out of 400 separate databases without first consolidating all that information into a single database, NSA can pull details out of a wide range of telecoms and other databases WITHOUT needing to have their own copy.

    No dragnet required!

  23. orionATL says:

    @Greg Bean (@GregLBean):

    i don’t know how my comments are read by you since this is pure electronic communication – no voice inflexions, no facial muscle communication.

    what i was thinking when i read your post on available technologies for doing what nsa claims only it can do was:

    “wait a minute, here’s an electronic/internet communications expert who says what nsa is doing and claiming only it can do, is at present being done by other, non-government organizations (nyse).

    immediately, i say to myself nsa has been conning us citizens.

    immediately, i infer, rightly or wrongly, that a non-secret, non-governmental entity, e.g., a phone company or a quasi- private organization like the post office, could do everything “data collection and storage” that nsa is doing.

    the difference would be this entity would not be secret, would not have a secret budget, would not be allowed to hold secret congressional oversight seances, and could be made subject to privacy laws and, specifically, to a stringent interpretation of article IV.

    i don’t know if this clears up for you where i stand or not.

    i will add i have some experience in govt bureaucracy, mostly very positive experience working with competent bosses and colleagues, though none in intell or military.

    my sense of the nsa cancer/mushroom is that it grew in the dark feeding on bullshit.

    before sept 2001 there was a way to do policing that included regulated electronic spying. after that date, things began to change so that the constitution became merely “agod damned piece of paper” and law enforcement prosecution bureaucracies doj/fbi slyly tipped the balance of legal power vs personal privacy grossly in their favor.

    to sum up, i don’t believe for a minute that the current nsa behemoth is needed.

    i believe it needs to be split up.

    i doubt it the nsa is very effective; it is drowning in info like a hi-altitude climber with pulmonery embolism.

    it seems too stupidly, fixedly focused on collection and storage and on protecting itself from public exposure to function effectively as an intelligence analyst.

    back to where i began, “upstream”, as nsa would say.

    you provided today the first specific technological info i have come across which challenges nsa’s necessity as a vacuum cleaner and nearly infinite dust-bunny storage unit.

    thanks.

  24. Greg Bean (@GregLBean) says:

    @orionATL: My pleasure. Agree with your conclusion entirely. And back to where I began, it’s not a technological issue, it’s about the evil NSA empire and its immorality and lawlessness. East Germany did not miss the Stasi and the US won’t miss the NSA. Just the opposite, life will improve for all, worldwide.

  25. jerryy says:

    @Greg Bean (@GregLBean): This thread is about to leave the front page, but I am sure it will come back in a new form later….

    Re: comment #28, …”No dragnet required!”

    You have illustrated the definition of a dragnet.

    http://dictionary.reference.com/browse/dragnet
    2. a system or network for finding or catching someone, as a criminal wanted by the police.

    which is also how the ACLU uses the term:
    https://www.aclu.org/files/images/nsaspying/asset_upload_file578_35950.pdf

    Having the sources (big data type of database) stored by other folks (non-local copy) just means someone else foots the storage cost bill, you have to sweep your net through their data to get what you want.

  26. orionATL says:

    @jerryy:

    yes, but in the scheme i envision, law enforcement have to ask a non-secret court for a suspect person’s communications. fishing and false affadavits would be ruled unacceptable.

    there has to be some accomodation for law enforcement needs, but those needs/requests must not be allowed to pre-empt bill of rights protections, nor be made in secret, nor may they be kept from being evaluated by the public courts or defense attorneys, nor may they be kept in secret archives for any but the smallest possible duration.

    budgets and operating procedures must be open to public review.

    of greatest importance would be very, very strong whistleblowing/personnel protection policies. it’s the ethical folks on the inside of organizations that are the greatest source of information and criticism relative to abuse, corruption, mismanagement.

    i agree this is an important discussion to have with a lot more to be said by many interested and concerned folk.

    thanks for your contribution.

Comments are closed.