Are Escaped Zoo Animals Autonomous?

Back when David Sanger revealed new details of how StuxNet broke free of Natanz, he used the metaphor of an escaped zoo animal actively unlocking its cage.

In the summer of 2010, shortly after a new variant of the worm had been sent into Natanz, it became clear that the worm, which was never supposed to leave the Natanz machines, had broken free, like a zoo animal that found the keys to the cage. It fell to Mr. Panetta and two other crucial players in Olympic Games — General Cartwright, the vice chairman of the Joint Chiefs of Staff, and Michael J. Morell, the deputy director of the C.I.A. — to break the news to Mr. Obama and Mr. Biden.

An error in the code, they said, had led it to spread to an engineer’s computer when it was hooked up to the centrifuges. When the engineer left Natanz and connected the computer to the Internet, the American- and Israeli-made bug failed to recognize that its environment had changed. It began replicating itself all around the world. [my emphasis]

This zoo animal found the keys to its cage, broke free, spread to an engineer’s computer, failed to recognize its new environment, and then began replicating itself all around the world.

That is, Sanger used the language of a cognizant being, acting as an agent to spread itself. That’s not inapt. After all, viruses do spread themselves (though they don’t actually go seek out keys to do so).

Which is why this detail, noted in Obama’s other pre-Thanksgiving document dump, is so stunning. (h/t Trevor Timm)

The Defense Department does not require developers of computer systems that launch cyber operations to implement the same safeguards required of traditional arms makers to prevent collateral damage.

[snip]

directive, released Nov. 21, mandated that automated and semi-autonomous weaponry — such as guided munitions that independently select targets — must have human machine interfaces and “be designed to allow commanders and operators to exercise appropriate levels of human judgment over the use of force.” The mandate called for “rigorous hardware and software verification and validation” to ensure that engagements could be terminated if not completed in a designated time frame. The goal is to minimize “unintended engagements,” the document states.

The Pentagon is permitting less human control over systems that deploy malware, exploits and mitigation tools, highlighting Defense’s focus on agile responses to computer threats. The document, signed by Deputy Secretary of Defense Ashton Carter, explicitly states that the directive “does not apply to autonomous or semi-autonomous cyberspace systems for cyberspace operations.”

We have already lost control of one our semi-autonomous cyberspace operations. The potential danger from its “escape” could be tremendous.

And yet DOD specifically exempts similar operations in the future? So we can commit the same error again?

Tweet about this on Twitter4Share on Reddit0Share on Facebook4Google+0Email to someone

15 Responses to Are Escaped Zoo Animals Autonomous?

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15

Emptywheel Twitterverse
emptywheel @joshleitzel Not a lot. One of the most interesting details is the way OLC memos point to national emergency rather than AUMF.
45mreplyretweetfavorite
bmaz My question at the outset was why GM concealment was not bankruptcy fraud; now that will be litigated. Good. http://t.co/CCL3wm2HYE
8hreplyretweetfavorite
bmaz @trevortimm Be terrified. Very terrified. Cause what you saw is, I think, all you get.
9hreplyretweetfavorite
bmaz @johnson_carrie According to my wife, "impossible jerk" characterizes lawyers in many locales @npratc
9hreplyretweetfavorite
bmaz @HoltenMark @mucha_carlos @ColMorrisDavis @KenDilanianLAT The constitutional framing is amazingly resilient, but resets are slow.
9hreplyretweetfavorite
bmaz @HoltenMark @mucha_carlos @ColMorrisDavis @KenDilanianLAT I represent far too many of the former and lament the latter. Things change though
9hreplyretweetfavorite
bmaz @HoltenMark @mucha_carlos @ColMorrisDavis @KenDilanianLAT Frankly, US can exert such influence, will not be effective foreign prosec either
9hreplyretweetfavorite
bmaz @HoltenMark @mucha_carlos @ColMorrisDavis @KenDilanianLAT Yes, in these considerations, that is exactly right. Not happening.
9hreplyretweetfavorite
bmaz @HoltenMark @mucha_carlos @ColMorrisDavis @KenDilanianLAT I wasn't being a smart ass, just honest as to situation.
10hreplyretweetfavorite
bmaz @mucha_carlos @ColMorrisDavis @KenDilanianLAT @HoltenMark Safe enough bet; no administration will want to open that can of worms.
10hreplyretweetfavorite
bmaz @mucha_carlos @ColMorrisDavis @KenDilanianLAT @HoltenMark ...ought to give pause in above regards too. If DOJ ever cared about these crimes.
10hreplyretweetfavorite
bmaz @mucha_carlos @ColMorrisDavis @KenDilanianLAT @HoltenMark Well, yes, and the wild expansion of extraterritorial jurisdiction in other cases
10hreplyretweetfavorite
November 2012
S M T W T F S
« Oct   Dec »
 123
45678910
11121314151617
18192021222324
252627282930