How Can NSA Protect Our Power Grid from Cyberattack When It Can’t Keep Its Own Power On?
In the United States, it is usually a safe bet to attribute massive government fuck-ups to the bloated contractors we’ve outsourced our projects to.
And the electrical problems plaguing NSA’s new UT data center — described as lightening in a box that has caused $100,000 of damage each of the 10 times it has happened — do seem to stem from poorly supervised contractors.
The Army Corps of Engineers is overseeing the data center’s construction. Chief of Construction Operations, Norbert Suter said, “the cause of the electrical issues was identified by the team, and is currently being corrected by the contractor.” He said the Corps would ensure the center is “completely reliable” before handing it over to the NSA.
But another government assessment concluded the contractor’s proposed solutions fall short and the causes of eight of the failures haven’t been conclusively determined. “We did not find any indication that the proposed equipment modification measures will be effective in preventing future incidents,” said a report last week by special investigators from the Army Corps of Engineers known as a Tiger Team.
It took six months for investigators to determine the causes of two of the failures. In the months that followed, the contractors employed more than 30 independent experts that conducted 160 tests over 50,000 man-hours, according to project documents.
Contractors have started installing devices that insulate the power system from a failure and would reduce damage to the electrical machinery. But the fix wouldn’t prevent the failures, according to project documents and current and former officials.
Now, don’t pee your pants laughing.
But I did have two thoughts as I read this.
First, this extended confusion sounds similar to that which Iranian nuclear scientists experienced as they tried to figure out why their centrifuges kept blowing up, thanks to StuxNet. While I think the chances some kind of hack caused this are small (but not zero), I do find it ironic that we cause ourselves the same kind of havoc we cause our worst enemies.
And consider the mission!
Back in February, Keith Alexander warned of the possibility of cyberattacks on our grid (which, anonymous sources made clear, could probably only be launched by China or Russia, but that didn’t stop Alexander from suggesting Anonymous might launch such attacks). The NSA needs more authority to protect against attacks that might bring down our power sources, the head of the NSA suggested.
But the entity that proposes to wield that authority, it seems, can’t even build a brand spanking new electrical system immune from some kind of failure.
It takes only one wiring mistake to create a shrapnel explosion. Miss wiring the phases from the emergency power supply for one and then “testing” it.
The Army corp of Engineers and their contractors have probably discovered a few other interesting methods as well.
As for blaming Iranians, who would have put such stuff under software control in the first place?
in a building that special who knows.
in ignorance, i’d bet on a serious, systemic grounding problem, perhaps associated with rebar in concrete. large-scale, high voltage wiring is both dangerous and tricky. perhaps particularly so in a building that may be laced with copper “spook” wiring to prevent the martians from learning there is unintelligent life on earth.
but of course the contractor knew that, didn’t it?
@greengiant: Oh, I definitely don’t think the Iranians did this. As I said, best bet is incompetent contractor.
The idea that cyberattacks on the grid could only come from China or Russia is ludicrous. Any competent developer with the right incentives could do it. Bringing down a major portion of the grid is easier than what I do on a day-to-day basis.
personally, i think it is most likely an “inductance attack” by brazilian poltergiest.
particularly if it involves those little fairly simple industrial computers – i forget how they’re denominated – that rayne has written about here which control so much industrial activity here (as with, also, the iranian centrifuge computers).
Thank goodness electricity isn’t all that important for a facility like this.
When malevolence and incompetence meet.
If these masters of the universe were any smarter they’d be dangerous.
For a country so paranoid about the compromising of the grid, there is absolutely no excuse to have a grid vulnerable by having IP addresses for operational hardware (I’m assuming that’s the case if they are paranoid).
The grid by definition is a well wired network that can operate independent of existing data/communications infrastructure. How difficult can it be to have separate network for communications/control of the grid by just modulating the data on the grid itself? Forget about IP addresses, just use proprietary formats.
Ha! I got this far and burst out laughing.
Because Katrina, the result of their engineering. And because
Okay, now I’ll continue reading
cf the hydraulic pumps the Army Corps of Engineers installed in New Orleans:
ok, reading on
@orionATL: Thanks for that, orionATL.
hmmm…is there Congressional oversight?
Except the govt approached the telecoms in February 2001 to do warrantless wiretapping, and the only one who said no was Joseph Nacchio at Qwest. So the beginning of the article (I just started reading) already implies “because terror,” because 9/11, which cannot be true.
Oh, fuck. This is stupid. Because “intelligence” backed by NSA compartmented walnut-shell paranoia is stupid. Bradley Manning was the one guy in his intelligence post in Iraq who surfed the web and tried to see a big picture, and everybody else in his unit with paranoid tunnel vision came to him to find out what was going on in the world, iirc from the trial.
I guess I should say mho. But the trouble with the paranoid IC is that nobody says mho, they have to say “I speak for all, and this is it.” At least by the time they report it anonymously and seriously to us. And if you do something that fucks their spin…
A WSJ blog that isn’t behind a paywall:
“the NSA has a history of poorly managed electrical projects.
In 2006, the agency found itself running out of electricity at its Fort Meade, Md., headquarters, was unable to plug in new supercomputers and resorted to shutting down “non-essential” uses of power. That problem had been understood for at least a decade but hadn’t been prioritized high enough to fix.
The problems flared again in the summer of 2007, when NSA had to resort to rolling brownouts and delayed the deployment of data-processing equipment to keep its systems running.
At the Utah Data Center, some engineers have suggested that the best way to fix the electrical failures would be to replace the problematic parts of the system, but the idea gained little traction when project officials realized it would cost hundreds of millions of dollars, a project official said.”
“The problem, and we all know it, is that they put the appliances too close together,” a person familar with the database construction told FORBES, describing the arcs as creating “kill zones.” “They used wiring that’s not adequate to the task. We all talked about the fact that it wasn’t going to work.”
More stories from local media:
“The Utah Data Center was scheduled to be operational last month or this month — the NSA would not specify — and employ about 200 people.
The Salt Lake Tribune has previously reported that the 1 million-square-foot data center will use 65 megawatts of electricity (a single megawatt is enough to run appliances in 500 to 750 homes). Rocky Mountain Power’s largest Utah customer is Kennecott Utah Copper, which uses about 200 megawatts.”
Who you gonna call?
It’s only Ringo: https://www.youtube.com/watch?v=mlH767uV1z8
thanks for the informative cites.
from the shape of the data center in the forbes article i conclude that the arc fault problem results from nsa/architects having accidentally built a cyclotron.
He’s leaving out plain old American constitutional democracy and the people of the state of Utah.
Most intriguing idea I’ve heard yet — using state’s rights to stop the NSA. Utah could turn off the NSA’s water spigot.
I got that from listening to Michael Boldin interview on The Scott Horton Show (the radio one, not The Other Scott Horton), who cited California Gov. Jerry Brown signing AB351 into law to refuse California’s participation in NDAA.
My governor! My left wing and right wing coalition! We can fly!
Let’s hope it never works. Maybe some idealistic techies working on the project have thrown a spanner (or several spanners) in the works?
Ad at the beginning of the program:
@lysias: Or Mother Nature. All that drilling and fracking in Utah — remember what Tim DeChristopher was trying to disrupt, in Utah? What could possibly go wrong with the computer center’s critical water supply? No techies or citizen votes required. Think Fukushima. One bad day.
@thatvisionthing: Another day, another window – craazyman on Yves Smith:
@thatvisionthing: What could possibly go wrong… Just occurred to me, has anyone checked to see if this thing that takes 1.7 million gallons of water a day to operate might be… nuclear powered?
Now I am thinking Homer Simpson.