The Chinese Turned Out My Lights (Maybe)
Remember that terrible blackout in 2003, that took power out from MI to NY and in between?
It was great fun here in Ann Arbor, for a little while. You could walk down the streets of the city and sushi merchants would come out and pretty much give their sushi away. We had an "apocalypse" barbecue that night, where everyone brought all the meat from their freezer or fridge and any alcohol that was cold, and consumed it in one big gluttonous barbecue. I had a non-electric land-line at the time and a gas stove and it was summer time, so I was pretty comfortable for the whole two-day affair. But it quickly turned our freeways heading west (where there was still power) into parking lots and those with electrical phones lost their communication and aside from the gluttony it was a big expensive mess.
Apparently, the Chinese did it.
Computer hackers in China, including those working on behalf of the Chinese government and military, have penetrated deeply into the information systems of U.S. companies and government agencies, stolen proprietary information from American executives in advance of their business meetings in China, and, in a few cases, gained access to electric power plants in the United States, possibly triggering two recent and widespread blackouts in Florida and the Northeast, according to U.S. government officials and computer-security experts.
One prominent expert told National Journal he believes that China’s People’s Liberation Army played a role in the power outages. Tim Bennett, the former president of the Cyber Security Industry Alliance, a leading trade group, said that U.S. intelligence officials have told him that the PLA in 2003 gained access to a network that controlled electric power systems serving the northeastern United States. The intelligence officials said that forensic analysis had confirmed the source, Bennett said. “They said that, with confidence, it had been traced back to the PLA.” These officials believe that the intrusion may have precipitated the largest blackout in North American history, which occurred in August of that year. A 9,300-square-mile area, touching Michigan, Ohio, New York, and parts of Canada, lost power; an estimated 50 million people were affected.
Officially, the blackout was attributed to a variety of factors, none of which involved foreign intervention. Investigators blamed “overgrown trees” that came into contact with strained high-voltage lines near facilities in Ohio owned by FirstEnergy Corp. More than 100 power plants were shut down during the cascading failure. A computer virus, then in wide circulation, disrupted the communications lines that utility companies use to manage the power grid, and this exacerbated the problem. The blackout prompted President Bush to address the nation the day it happened. Power was mostly restored within 24 hours.
Read the whole article, if only for the description of a Chinese hacker accidentally bringing down Florida’s grid while (s)he was trying to map it. Oops.
The whole thing reads to me like a great long advertisement for Richard Clarke’s new book, Your Government Failed You (Clarke is scheduled to do an FDL Book Salon in July). I heard Clarke on Fresh Air yesterday, and he basically argues that the Bush Administration has only just started preparing some kind of response to such cyber-attacks in the last year. Clarke said he had presented a plan to defend against such attacks in 2003, but Bush basically deep-sixed it (at least he’s consistent in rejecting Clarke’s good advice). Given the timing, and given how frequently Clarke mentioned the vulnerability of our nation’s power grid, I would be unsurprised if Bush asked for that report in response to the 2003 blackout.
Harris, too, describes the Administration’s stalling on responding to this threat.
President Bush has personally devoted more high-level attention to the cyberattack issue in the last year or so than he did in the first six years of his tenure combined. Many security experts are surprised that the administration is only now moving to take dramatic measures to improve the security of government networks, because some Cabinet-level and White House officials have been warning about the threat for years to just about anyone who will listen.
Until McConnell, the national intelligence director, personally drove the point home to Bush in an Oval Office meeting in 2006, there was little top-level support for a comprehensive government cyber-security plan. “They ignored it,” one former senior administration official said flatly. “McConnell has the president’s ear.”
(I would imagine Clarke is a top candidate to be that SAO.)
And yet, as we learned the other day, our belated efforts to respond have been plagued by the same kind of secretive paranoia with which Bush always functions.
All in all, this report looks like the kind of report you’d get from a very positive elementary school teacher. "Very nice try, Johnny. It’s so nice to see you trying to finish the homework you’ve been working on for eight years. Now let’s talk about the bare minimum you’re going to need to do in order to actually complete this homework. And no, you can’t have $17 billion dollars for what thus far is still C minus work."
All of which is a very disorganized way of saying you ought to make sure your disaster supplies will support you for longer than just one gluttonous apocalypse party, because you may well need them.
BushCo never seems to get even one rung up the ladder from Clarke’s initial assessment of them in Against All Enemies. Both then & still, incompetence coupled w/arrogance seems the only strong suit of the WH team. In his tv interviews Clarke always appears so calm, even patient in his detailing of what went wrong under BushCo. Remembering his testimony before Congress, one realizes that his calm & patience was bought @ the price of great pain.
All of which is a very disorganized way of saying you ought to make sure your disaster supplies will support you for longer than just one gluttonous apocalypse party, because you may well need them.
Living in earthquake prone SoCal, disaster supply stores are plentiful. We’ve even got the generator & solar water heater. Freeze dried food is fun to eat if you pretend you’re camping out, which we all may be @ some point, maybe during the next Big One. The state gov. here is pretty good @ responding to disasters- they never wait for the fed gov. to kick in. Probably a smart idea not to…
Marcy – I spotlighted this …and also sent it to my sister, who works for NASD. Considering that utilities will do things like try to save money by not doing their tree trimming, makes you wonder how much they are bothering to spend on network security at all.
Pull the other one!
My very handy, electrician husband (who is a conspiracy theorist and bonafide paranoid)predicts a world run by China before long. (maybe it’s the reality now) When we were young we had a friend of the family who was a pilot for the airforce out of sac.
He was too smart and spent too much time in foreign countries, drank way too much during peace time, to be “just a pilot”. This was during the 70’s in the years after vietnam. When he would get drunk, he always stayed “smart”, he would tell us, “it’s not the soviets who we need to worry about, they are going to destroy themselves, it is the chinese.” I remembered him and his words because he told us that the soviets would soon destroy themselves. (and Reagan hadn’t begun to roar, yet) And of course we all grew up with the notion of “The Sleeping Giant”.
But when you compare population, resources, intelligence and education, the value of discipline, the loans out to the U.S, their need for energy, and the explosion of capitalism taking place in China…how can it not be the future. Hope would be that we can synthesize the polars between their system and ours to something liveable. Clearly capitalism has won over there, but we all know what you get when you join capitalism with authoritarianism.
While my husband sees the glass half empty, I see potential for a glass half full. But China is coming to America, whether we like it or not.
Ah the yellow menace raises its ugly head again. Can’t trust those Chi Comms, unless you’re doing business with them of course. While I agree anything this Administration touches turns to merde, you’d think the geeks at NSA would have had a handle on the hacks, if indeed they occurred. Also, IT geeks at the power companies should have noticed something was afoot.
Having consulted for and worked for more than one energy company, I would doubt very much that the IT geeks would have noticed at the time. On the other hand, I’m extremely sceptical of the 2003 part of this story. The Florida part of the story sounds much more believable.
The fundamental problem that our government has is that the best steps to take to ward off these attacks would also making harder for our government to spy on us. You can guess which side of the fence our current overlords chose.
Which brings you back to the complaints of the SASC–that they’re trying to push through domestic spying in the guise of cyber-security.
As I recall in those particular incidents, the outages were due to neglect in portions of the grid owing ultimately to intrusions of hedge funds who have this vexing tendency to drain the funds of a firm and leave behind a broken shell.
There have been hackers in various systems, back into the 80s and 90s, and many of them were from outside the US. (I recommend Cuckoo’s Egg by Clifford Stoll, which covers one of the earlier incidents.) If Shrub is just getting around to dealing with it, he’s so far behind the curve that he can’t even see the curve.
On emergency supplies:
I figure to get some cases of full-meal MREs, with heaters. I’m told by One Who Knows that a full-meal MRE runs about 3500 calories, thus at least a day’s worth of calories. I’m also told that they tend to be binding, so more water may be a good idea.
Canned food is good, too – but not powdered peanut butter!
(Some of my fellow-commuters figure we really need a three months’ supply of food and water in each household.)
I like reading thes kinds of books. Thanks for the headsup.
The paradox of the Chinese who are outstriping us technology wise, but can’t build schools that don’t collapse and kill their children is ironic.
DHS/US cybersecurity is a myth. I’d expect many more attacks on US cities via the porous mythical defenses.
Here’s a general overview of leveraged private equity firms (what I referred to earlier as hedge funds) and their impacts on the integrity of going concerns they acquire and leverage to death: Pirates of Private Equity.
At rock bottom, it’s this ability to borrow the way they do that shouldn’t be possible in a rational world.
Whew, finally Canada is off the hook … *g*
It seems like we’re getting two warm weekends in a row up here, am off to do some gardening, have a great weekend !
Anyone else get the feeling that Somebody has noticed this post? (Iow, anyone else getting lots of blips when connecting?)
Petrocelli, was Canada ever on the hook? I didn’t know that was a theory. We actually flew out of Toronto that day about two hours before the power failure; we heard about it on the radio driving into town from the Calgary airport. I was worried about my husband, who had been in a nursing home for a little more than a month — that, of course, turned out to be the best place to be, since they had back-up generators and became a community centre for a couple of days — air conditioning, land lines, etc. My cat sitter phone-napped one of my sets because she and her roomies only had cells, and their power didn’t come back for several days, although I think that my place was up and running by the next morning.
I also flew out of Canada on the day the War Measures Act was lowered in 1970 — missed the whole (outrageous) martial-law thing. I don’t travel as often as EW does, but I promise to let you know if I’m taking off again any time soon.
The first ‘theory’ on CNN & Friends was that something happened in Ontario that caused the entire system to trip. Mayor Mel Lastman blew his fuse during a CNN interview, uttering this infamous line, “When have you ever heard the United States take the blame for anything? This is no different.”
The U.S. MSM reacted with pseudo- patriotic outrage, about those “Arrogant Canadians who live in self- denial” … which prompted a country wide laughing fit that lasted until the start of Hockey season … *g*
A short while later, the “problem” was traced to Ohio …
Please do not leave Canada again, the nation resorts to self- destructive behavior, it appears … *g*
BTW, did anyone speak to Ishmael about a “Drinking Liberally”meetup in Toronto ? Marcy says she would attend.
Speaking of the Chinese, via laura Rozen,Gertz subpoenaed to fork over his sources on a Chinese espionage story in 2006.
ah, memories of the california rolling-blackouts courtesy bushco’s energy buds gaming a flawed system. hospitals forced onto generators as our reps were turned back at the white house door, the investigations put into cold storage post-911.
at the ready in my kitchen my earthquake kit, two weeks of supplies. no bbq. may have to add that.
Chinese, espionage and computers seem to be the theme today…
http://www.telegraph.co.uk/new…..aptop.html
A report based entirely on anonymous, even secondhand, “US officials”? Count me skeptical.
me too.
how sure are we this isn’t a product of Team B paranoia?
not saying it’s not true, or even if false that we shouldn’t be thinking about security in this realm. i just need more to move from a state of agnostic skepticism.
There’s been a brouhaha recently over whether the Chinese had unauthorized access to a Commerce Department laptop taken to China in connection with the Commerce Sec’y’s recent visit. If they did, they not only directly copied valuable data on the disk, but obtained information on the Commerce Department’s access codes, protocols, etc. I’d call that pouring a bottle of salt into a salt mine.
Our commercial dependency on the Chinese has mushroomed in the past ten years and creates risks most companies work hard to ignore because they absolutely, positively have to be there, have to outsource there, have to work with them. That’s also true of the USG: China is the happy lender supporting a big chunk of Overspendin’ George’s big overdraft.
It is true that the Chinese are an essential player in world commerce. A small part of that is via its loans and investment capital. They manufacture many of the world’s white goods (washers, dryers, appliances) and will soon manufacture more cars than the US (GM lost $38 billion last year and $13 billion in the first quarter this year; it’s also shedding more than 20,000 workers again this year). China is fast becoming the top outsourcing location. If you’ve outsourced anything, your supplier or its suppliers has ops there, and most likely chunks of your data.
Working with China with one’s eyes wide shut would be an act of self-immolation. In the places Americans travel, its Eastern seaboard, major cities inland, top and middling hotels, airports, etc., electronic and physical surveillance is as routine as in central London, where the average tourist is photographed well over 3,000 times in a two-week stay.
Check into a major hotel and within seconds state security knows your room number, passport and visa details and history, travel arrangements, prior visits, etc. If you’re an interesting person or represent an interesting player, they listen in to phone, internet and local conversations. Be foolish enough to discuss important details about a meeting and by next day, your counterparts will have a transliterated transcript with all the details. Occasionally, they may be so self-satisfied with how easy it all was that they quote back to you some of what you said or e-mailed to your boss, lawyer or finance gal the night before. Your bottom line price, for example.
The Chinese had that capability years ago. Now, of course, you needn’t visit Beijing or Shanghai to get that treatment. Not if your data or servers are in China, you have key operations are in China, or your critical supplies are sourced in China. If Bush were as smart as Cheney tells him he is, he’d be working a lot harder on counter-espionage and a lot less on copying your e-mails and phone calls to your granny or new lover. But if he were that smart, he’d be president instead of Dick Cheney. I wonder who the White House and its outsourced telecoms providers work with in China?
Here’s my paranoia.
I see the shoddy nature of just about every single item imported from China as an intentional move on the part of the Chinese. We may have invented ‘planned obsolescence’, but we are now being taught by some very, very quick learners. I am always ready to be pleasantly surprized if something “Made in China” lasts as long as it ought to — but I have yet to have that pleasurable experience.
I have found contacts within appliances snipped and bent so as to cause arcing and premature failure, for example. (The “Made in Poland” replacement from another company serves reliably for years and is still going.) I have bought screws ‘Made in China’ that did not even have the SLOTS in their heads, for pity sakes. (I keep a few packages of Swedish bronze screws around just to preserve the memory of what a screw should look like.)
If the spigot of Chinese goods is shut off, the early failure of the Chinese goods in use is going to knock us “back to the Stone Age”, and we are not going to be able to make our own replacements — the factories and skills have gone.
Causing domestic chaos without overt triggering actions whether slowly, as I describe, or quickly, as this post suggests, is the perfect method of warfare. Why, we even pay them for the privilege.
The Chinese are quite capable of producing world class goods and services. Getting them here confronts two problems. Their customers’ willingness to pay for them — in many cases, they’d rather have cheaper goods and profit, or more profit — and China now keeps much of the best that it produces for itself, like the Swiss, who get first tibs on Swiss equipment, machinery, wines, etc. The gnomes, however, have a much smaller market, about the size of Shanghai to satisfy.
I would be cautious about characterizing the whole spear by the knotty attributes of its shaft. The point might still be marvelously shaped and sharp.
American ethnicity, or its technical or financial prowess can’t be adequately defined by the attributes of Appalachia, Grosse Point or SE Washington, DC. Neither can the capabilities of China be adequately judged by its washers & dryers, Beijing traffic jams or the local fast-food version of “Chinese” cuisine. Though they would welcome much of America underestimating them.
Gee – the chinese are behind it all? Does that mean that the dominent theory – that “privitization” of the national power grid, under-maintenence of equipment, variable competence of regional powerco’s — is all wrong?
Before letting the entrepreneurial mess of our power supply off the hook, and blaming it all on the damn yellow Chinaman, I’d personally like to see some evidence.
Somebody clue me in as to why the Chinese would want to disrupt a large swath of the US? Or are you saying it was an accident? While I’m certainly in favor of increased security against hackers for our infrastructure, its not clear to me whether the larger threat is malicious intent, or general incompetence.
The Chinese may have read John Perkins (Secrets of an Economic Hit Man & The Secret History of American Empire) and Naomi Klein’s The Shock Doctrine and seem to be putting those theories to some use. The next president, no thanks to the shallow thinkers in this administration, will be under their thumb, or more to the point we will be without knowing it. What a disaster, thank you Wall Street and its many enablers.
Unlike many of the topics of discussion here on EW on which I am only marginally informed (a fact that may be groaningly obvious to readers of some my comments on them), this is one with which I am quite familiar having worked for over 40 years for either suppliers of power system control technology to the electric utility industry, or consultants to the industry on the subject. This is the first I’ve heard of the possibility of foreign involvement in blackouts, although admittedly my ear is not as close to the ground as it used to be.
Basically, I’m with Professor Foland @ 18; I’ll have to see a lot more specific information directly from sources close to the relavent specialists in the industry before attributing the Northeast blackout of August, 2003, and the one in Florida in February of this year to specifically identifiable, malicious action on anyone’s part, Chinese or otherwise. I won’t rule it out, either, however. In order to accomplish this the malefactor would have to not only penetrate the Energy Management System (EMS) that manages one (or several) of the larger of the grid’s balancing areas (formerly called “control areas”), but s/he would have to have enough knowledge of the configuration of the power system they were trying to take down in order to attack its most vulnerable points.
Prior to the mid 1990s, most EMSs were stand-alone units not connected to anything but the remote terminals at the substations and power plants it monitored and controlled, to the EMSs of the neighboring utilities and to the power pool of which it was a part. They also, in most cases, were not systems whose hardware and software was provided by the vendors who supplied the companies’ IT shops. In fact, most IT shops didn’t want anything to do with System Operations, since the latter were 24/7 operations. The corporate IT guys had no desire to have their asses dragged out of bed at 3:00 am on Sunday morning for phone consultation because the EMS crashed, or worse yet have to drive to the control center to restart it. Besides, the data being captured was not time-critical for anyone except the power system dispatchers. Daily and monthly reports were more than good enough for business purposes.
One of the consequences of “deregulation” (don’t get me started) is that it made real-time EMS information enterprise-critical, and that meant connecting the EMS to the corporate network. This began to occur big time in the mid-1990s, and especially in the run-up to the Y2K changeover. This was also a period in which vendors were compelled by competitive pressure to switch to widely applicable hardware platforms such as engineering workstations and servers running UNIX (and later Windows NT) instead of computer hardware and operating systems custom designed for the real-time market niche. And with those platforms the industry also adopted some multi-purpose, shrink-wrapped software, such as relational data base managers like Oracle.
The utilities I am most familiar with all approached this with great caution, and adopted the best technology available at the time. In all cases I’m aware of, the EMS interfaces to the corporate networks were protected with firewalls, and those enterprise networks were similarly protected from the public internet. Thus, the only way they could be penetrated from outside the company was to go through first the corporate firewall and then the local, EMS one. That’s not to say they were invulnerable, but penetrating them would have been a challenge.
I recall reading an engineering analysis of the 2003 Northeast incident that said a major factor in the chain of events was that the First Energy EMS was down (both primary and back-up) and the resultant reduced situational awareness of the dispatchers led to some inappropriate decisions on their part. IIRC, viruses were viewed as a probable cause of the EMS outage. That virus may well have had a Chinese origin. Or it may not. In any case I’ll have to see some evidence before concluding that it was planted in that computer for the express purpose of taking down the grid.
Regarding the Florida event, the NJ article jumped off from the fact that eight power plants went down to imply some sort of foul play. I’ll have to see a lot more than that, however, to begin to suspect that. Because Florida is a penninsula it is interconnected with the rest of the grid on only its northern borders. Furthermore, the largest load centers are in the far south. Whenever possible, which is almost all of the time, a power system is operated with sufficient spinning and other generation reserves such that it can survive the worst projected adverse event (loss of the largest of its generators, trasnmission lines, etc.) with no loss of load. The malfunctioning equipment is automatically disconnected and life goes on.
Occasionally things don’t work as planned, however, or out of necessity the power system is being operated too close to the edge, and the power system becomes unstable enough to break up into islands because the protection systems detect overload conditions that will cause bad things to happen to expensive, not readily replaceable equipment. Some of these islands are isolated from all sources of supply and thus go immediately black. Other islands include an amount of generation on line that is close enough to the total load in the area such that stability can be maintained in that island. Its lights usuall stay on through the duration of thre blackout. Other islands, however, have imbalances between generation and load that are too large to allow a speedy restoration of stability. Whether there is an excess or a deficit of generation, the generators are vulnerable to severe damage in either case. After a few seconds or minutes the vulnerable generators will automatically trip off and this area, too, will go dark. This is how blackouts happen, and Florida’s geography makes the state more vulnerable than most. Unless and until I see a detailed engineering analysis that irrefutably points to malicious hacking, I’ll put my money that it was caused by some scenario such as described above.
PS: I know, I’m late to the party. I’ve been writing this intermittantly over the course of several hours while visiting my daughter’s family and meeting my first grandchild, an experience that seriously undermines intellectual cohesion.
Over the weekend, a couple server admin friends were talking about the strange places they get hits from, and musing whether there’s some amorphous New Economic Rule emerging that could be called: The Logarithmic Increase in the Costs for Assholeness on a Globally Connected Planet. How many targeted attacks can a good reputation avert? Dunno, but why make yourself a target by being an asshole?
Unfortunately, I doubt BushCheney will examine this aspect of cyber security. More’s the pity.
For those interested, this interview with Clarke at MSNBC is worth a view; he argues that the politicization of National Security simply leads to bureaucratic dysfunction and political incompetence: http://www.msnbc.msn.com/id/21…..1#24892941
Looks like I’ll be adding his book to McClellan’s when I swing by my local bookseller in coming days.
—————-
MNChuck: Hope your trip is marvelous, and it’s wonderful you get to meet The New Gal in the family so soon.