Rattled: China’s Hardware Hack – PRC’s Response

[NB: Note the byline. Portions of my content are speculative. / ~Rayne]

The following analysis includes a copy of an initial response Bloomberg Businessweek received from the Ministry of Foreign Affairs for the People’s Republic of China (PRC) in response to its story, The Big Hack. In tandem with the Bloomberg story this was published on October 4 at this link. PRC’s response is offset in blockquote format. No signer was indicated in the published response. Additional responses to Bloomberg’s story will be posted separately.
__________

People’s Republic of China

China is a resolute defender of cybersecurity.[1] It advocates for the international community to work together on tackling cybersecurity threats through dialogue on the basis of mutual respect, equality and mutual benefit.

[1] It’s hard to argue that PRC does not defend its own cybersecurity resolutely.

[2] There are four themes here, at least:

— collaboration and ongoing dialog, but this requires trust which are difficult to develop without openness;
— mutuality, which again requires trust;
— equality, an insistence that footing of those in dialog is level;
— benefit, implying a transactional nature.

This may be a very small paragraph but it is heavily loaded and not for the kind of lightweight, half-assed diplomacy we’ve seen from this administration.

Supply chain safety in cyberspace is an issue of common concern, and China is also a victim.[3] China, Russia, and other member states of the Shanghai Cooperation Organization proposed an “International code of conduct for information security” to the United Nations as early as 2011.[4] It included a pledge to ensure the supply chain security of information and communications technology products and services, in order to prevent other states from using their advantages in resources and technologies to undermine the interest of other countries.[5] We hope parties make less gratuitous accusations and suspicions but conduct more constructive talk and collaboration so that we can work together in building a peaceful, safe, open, cooperative and orderly cyberspace.[6] —Translated by Bloomberg News in Beijing[7]

[3] What is PRC alleging here? Are they accusing the U.S. of compromising their supply chain? Difficult for the American public to debate this when it is so opaque though this comment may be based directly on NSA interception of networking equipment to be used in China as one example.
[4] What was happening between U.S. and Russia at that point in time? PRC acts as if an agreement to this code would happen in a vacuum.
[5] A dig at U.S.
[6] Another dig at U.S.
[7] There has been no apparent demand for correction to any of this translation.

Like Supermicro’s response this one is very short and effective, giving little away.

Still Rattled: Fallout and Pushback

[NB: Note the byline. Portions of this post may be speculative. / ~Rayne]

The tech industry and technology journalism outlets remain rattled by Bloomberg Businessweek’s The Big Hack article.

Bloomberg Businessweek’s Jordan Robertson and Michael Riley published a second article last Tuesday in which a security expert went on the record about compromised servers with Supermicro motherboards in an unnamed telecommunications provider. Do read the article; the timing of the discovery of the unexpected network communications and the off-spec covert chip fit within the timeline of Apple and Amazon problems with Supermicro motherboards.

The FBI’s and DHS’ responses are also interesting — the first refused to comment and the second offered a tepid endorsement of Apple’s and Amazon’s denials.

The second article hasn’t assuaged industry members or journalists, though, in spite of a source on the record about a third affected entity.

The main criticisms of Bloomberg piece are:

— No affected equipment or firmware has been produced for review;

— Too much of Bloomberg’s sourcing remains anonymous;

— The claims cannot be validated by other journalists, technology companies, persons at Apple and Amazon who have been contacted and interviewed by non-Bloomberg journalists;

— Contacts inside the companies in question continue to deny knowledge if they don’t express confusion about the alleged hack;

— Apple and Amazon have published firm denials, including Apple’s preemptive letter to Congress.

However,

— Something drove both Apple and Amazon to change their relationship with Supermicro within a fairly tight time frame;

— The uniformity of their early denials in which they avoid mentioning hardware and lean toward web application as a point of conflict is odd;

— Neither of these enormous firms nor Supermicro have filed a lawsuit against Bloomberg for libel that the public can see, preventing questioning of Bloomberg’s journalists and sources under subpoena;

— Securities and Exchange Commission doesn’t appear to have been engaged to investigate the claims (although it’s possible the SEC is on this and may simply not have disclosed this publicly);

— None of the other unnamed companies alleged to have received compromised motherboards have uttered a peep to defend (or rebut) Apple or Amazon.

I have not seen in any reporting I’ve read to date — from either Bloomberg Businessweek in The Big Hack or subsequent articles examining the claims or rebutting them — that any journalist, tech industry member or infosecurity community member has asked whether Apple, Amazon, or the other affected companies ordered customized motherboards or servers with customized motherboards made to their company’s specifications. Supermicro has also said nothing about any possible differentiation between motherboards for different companies which would affect the scenario. The silence on this point is confounding.

This piece in Ars Technica captures many of the concerns other tech news outlets have with the Bloomberg reports. Complaints that software — meaning firmware — is easier to hack than adding off-spec hardware miss two key points.

Made-to-order components or assemblies in Just-In-Time lean manufacturing enterprises make it easier to ensure that adulterated products reach their intended mark because each order represents an identified, traceable batch. Adherence to ISO standards in manufacturing processes may even make traceability easier.

We know Supermicro uses lean manufacturing techniques because it’s in job postings online (lousy pay, by the way, which may also say something).

Does Supermicro use the same lean manufacturing approach overseas? Do any of its suppliers also use lean manufacturing?

In contrast, release of firmware (without corresponding adulterated hardware) to a single target is more difficult to control than hardware — the example given is Stuxnet (excerpt here from Ars Technica).

Why wouldn’t a determined nation-state ensure there was a failover, a Plan B method for accessing specific intelligence from a narrow range of sources instead of betting the farm on one method alone? Given the means to deploy both malicious firmware and adulterated hardware, why wouldn’t they try both?

~ | ~ | ~

In spite of tech industry and journalists’ criticisms of Bloomberg’s reporting, these facts remain:

1 — Technology supply chain has been compromised;

2 — U.S. government has known about it (pdf);

3 — U.S. government has not been forthcoming about it or the blacklists it has implemented;

4 — U.S. government has tried to investigate the compromise but with insufficient success;

5 — Some companies are also aware of the compromised supply chain.

We’re no closer to resolving this question: has the compromise of the supply chain remained limited to counterfeiting, or does the compromise now include altered products?

At what point will the tech industry and infosecurity community begin to take supply chain hacks more seriously?

_________

[AN: I still have to analyze both Apple’s letter to Congress and its second response posted on their website along with Amazon’s published response. More to come./~Rayne]

Before Trump Did Nothing When Mohammed bin Salman Went After Jamal Khashoggi, He Did Nothing When MBS Went After Alwaleed bin Talal

There are a number of stories suggesting that the Trump administration will do nothing in response to the evidence that Mohammed bin Salman lured journalist Jamal Khashoggi to the Saudi consulate in Turkey to have him murdered and dismembered.

Trump has made a show of pretending to get to the bottom of things, while saying doing anything about it would hurt US-Saudi relations (meaning arms sales).

As outrage started to grow, MBS called Jared Kushner, with whom he has a close relationship sealed over all night conversations.

The White House said Wednesday that the powerful Saudi crown prince, Mohammed bin Salman, had spoken about Khashoggi the previous day with White House national security adviser John Bolton and Trump’s son-in-law and senior adviser Jared Kushner. Kushner and the crown prince, who is commonly referred to as MBS, are known to be close.

A former administration official told POLITICO that MBS had demanded the call earlier in the week after the top official at the U.S. Embassy in Riyadh asked MBS directly about the Khashoggi case. The crown prince denied any wrongdoing in his conversation with that embassy official, the former official said.

Neither the White House nor the State Department would comment on the Saudi crown prince’s demand or most other aspects of this story. But the former official said the crown prince’s insistence on talking directly to the White House indicates he is hoping to leverage his close ties with Kushner and others in Trump’s inner circle to avoid repercussions.

And the business community — including close Trump allies — seem prepared to head for an investors conference in Saudi Arabia in spite of the assassination.

But if it becomes clear that the prince ordered the assassination of Mr. Khashoggi or was connected to it in some way, it will provoke an outcry on Capitol Hill; embarrass American executives, dozens of whom are flocking to Riyadh for a conference next week where the crown prince is scheduled to speak; and put Mr. Kushner, who was once himself a newspaper publisher, in an extremely awkward position.

Among the prominent figures scheduled to take part are Jamie Dimon, the chief executive of JPMorgan Chase; Stephen A. Schwarzman, the chief executive of the Blackstone Group; and Dara Khosrowshahi, the chief executive of Uber.

Two other scheduled attendees have ties to Mr. Trump: Thomas J. Barrack Jr., a financier who is a friend of the president’s; and Dina H. Powell, a Goldman Sachs executive and former deputy national security adviser who worked closely with Mr. Kushner on Saudi Arabia and is a leading candidate to replace Nikki R. Haley as ambassador to the United Nations.

The Treasury Department said Mr. Mnuchin was still planning to attend.

While Congress has responded to this assassination by leveraging the Magnitsky Act, it seems the Administration would just like attention to the killing to fade.

Which really shouldn’t be a surprise.

The Administration did nothing last year when MBS targeted an even more prominent western-connected Saudi, Alwaleed bin Talal. Alwaleed was detained for 83 days by MBS until such time as he agreed to some kind of deal with the government, which may have involved handing over a substantial part of his fortune and acceptance of greater involvement in his business decisions.

Did you have to pay the government any money, did you have to hand over any land, did you have to surrender any shares?

When I say it’s a confidential and secret agreement, an arrangement based on a confirmed understanding between me and the government of Saudi Arabia, you have to respect that.

I’m a Saudi citizen. But I’m also a member of the royal family. The king is my uncle. Mohammed bin Salman is my cousin. So my interest is in maintaining the relationship between us and keeping it unscratched.

While Alwaleed is in no way a Saudi dissident, as Khashoggi was, he was a crucial cog not only in Saudi-US relations, but by virtue of his substantial investments in key US companies, in the US economy.

And western observers watched as MBS exerted some kind of influence over Alwaleed with only hushed complaints.

Far from criticizing the crackdown, Trump (and Jared, before the fact) appeared to sanction it.

Trump might do so not just because he has a fondness for authoritarianism. Starting fairly early in his presidential campaign, Trump had responded to Alwaleed’s criticisms of him with public mockery.

The Alwaleed-Trump tiff began in 2015, when candidate Trump called for curbing Muslim travel to the US in a bid to prevent terrorist attacks. Because of that, Alwaleed tweeted that the Republican front-runner was a “disgrace” and should bow out of the race. Mr. Trump responded that the prince was “dopey” and was seeking to “control our US politicians with daddy’s money.”

At one point, the future president tweeted a photo of Alwaleed alongside Megyn Kelly, then a Fox News correspondent who had clashed with Mr. Trump. It turned out that the image was a fake, and Mr. Trump falsely claimed that Alwaleed was “the co-owner of Fox News.” In fact, the prince had a stake in Fox’s (FOXA) sibling company, News Corp. (NWS), amounting to about 7 percent. He since has cut it drastically.

Alwaleed has countered Mr. Trump’s attacks by pointing out that he helped bail out the New York developer when the highly indebted Trump empire teetered on collapse in the early 1990s. First, the prince bought Mr. Trump’s 283-foot yacht for a bargain price of $18 million and with a partner bought out the Plaza, a storied New York hotel, which the Trump Organization owned.

Indeed, the Intercept reported that Jared provided intelligence from the Presidential Daily Brief to MBS on people he deemed disloyal to the regime.

In late October, Jared Kushner made an unannounced trip to Riyadh, catching some intelligence officials off guard. “The two princes are said to have stayed up until nearly 4 a.m. several nights, swapping stories and planning strategy,” the Washington Post’s David Ignatius reported at the time.

What exactly Kushner and the Saudi royal talked about in Riyadh may be known only to them, but after the meeting, Crown Prince Mohammed told confidants that Kushner had discussed the names of Saudis disloyal to the crown prince, according to three sources who have been in contact with members of the Saudi and Emirati royal families since the crackdown. Kushner, through his attorney’s spokesperson, denies having done so.

“Some questions by the media are so obviously false and ridiculous that they merit no response. This is one. The Intercept should know better,” said Peter Mirijanian, a spokesperson for Kushner’s lawyer Abbe Lowell.

On November 4, a week after Kushner returned to the U.S., the crown prince, known in official Washington by his initials MBS, launched what he called an anti-corruption crackdown. The Saudi government arrested dozens of members of the Saudi royal family and imprisoned them in the Ritz-Carlton Riyadh, which was first reported in English by The Intercept. The Saudi figures named in the President’s Daily Brief were among those rounded up; at least one was reportedly tortured.

While that story line of Trump’s response to the persecution was largely dropped as Alwaleed’s detention drew on early this year, I don’t doubt that Trump’s personal animosity to Alwaleed made him, if anything, at least comfortable if not enthusiastic about MBS’s power grab at Alwaleed’s expense. If so, MBS would have played to Trump’s own penchant for revenge to undercut what otherwise might have been more vocal criticism of the arbitrary treatment of a key international businessman (that said, the US made surprisingly little noise when MBS sidelined Mohammed bin Nayef, either).

And at that moment, MBS established that Trump would not interfere with any crackdown on opposition — because Trump has already bought into it.

Rattled: China’s Hardware Hack – SMCI’s Response

[NB: Note the byline. Portions of my content are speculative. / ~Rayne]

The following analysis includes a copy of an initial response Bloomberg Businessweek received from Super Micro Computer in response to its story, The Big Hack. In tandem with the Bloomberg story this was published on October 4 at this link. Super Micro Computer’s response is offset in blockquote format. No signer was indicated in the published response. Additional responses to Bloomberg’s story will be posted separately.
__________

Supermicro

While we would cooperate with any government investigation, we are not aware of any investigation regarding this topic nor have we been contacted by any government agency in this regard.[1] We are not aware of any customer dropping Supermicro as a supplier for this type of issue.[2]

[1] (a) “we are not aware” “nor have we been contacted” — who is we?

(b) “nor have we been contacted by any government agency” — has Supermicro been contacted by customers or their auditors or their security teams, contract or not, about security problems?

[2] Were one or more of Supermicro’s customers dropped by their customers because of security concerns including problems with firmware? Are any of the customers or customers of customers U.S. government entities?

Every major corporation in today’s security climate is constantly responding to threats and evolving their security posture. As part of that effort we are in regular contact with a variety of vendors, industry partners and government agencies sharing information on threats, best practices and new tools. This is standard practice in the industry today. However, we have not been in contact with any government agency regarding the issues you raised.[3]

[3] Has Supermicro been in contact with any government agency regarding any security issues including firmware updates?

Furthermore, Supermicro doesn’t design or manufacture networking chips or the associated firmware and we, as well as other leading server/storage companies, procure them from the same leading networking companies.[4]

[4] Interesting pointer about networking chips. What other motherboard content does Supermicro not design or manufacture, procuring from other companies? What procured motherboard components have firmware associated with them?

Rattled: China’s Hardware Hack – Amazon’s Response

[NB: Note the byline. Portions of my analysis may be speculative. / ~Rayne]

The following analysis includes a copy of an initial response  received from Amazon by Bloomberg Businessweek in response to its story, The Big Hack. In tandem with the Bloomberg story Amazon’s response was published on October 4 at this link. The text of Amazon’s response is offset in blockquote format. No signer was indicated in the published response. Additional responses by Amazon to Bloomberg’s story will be assessed separately in a future post.

This analysis is a work in progress and subject to change.
__________

Amazon

It’s untrue that AWS[1] knew about a supply chain compromise, an issue with malicious chips, or hardware modifications[2] when acquiring Elemental. It’s also untrue that AWS knew about servers containing malicious chips or modifications in data centers based in China, or that AWS worked with the FBI[3] to investigate or provide data about malicious hardware.

[1] Identity – were there ever any third-party contractors or representatives involved in the relationship with Elemental? With Supermicro? Are there more than one Amazon subsidiary entity involved in the evaluation, purchasing, implementation of Elemental or Supermicro products into Amazon or its subsidiary enterprise? Which entity submitted this denial to Bloomberg Businessweek: Amazon, AWS, or some other subsidiary?

[2] What about evidence of bad or mismatched firmware and firmware updates?

[3] Did any law enforcement, military, or intelligence agency work with Amazon or any of its subsidiaries or contractors to investigate or provide data on hardware which failed to operate to specification or as expected?

We’ve re-reviewed our records[4] relating to the Elemental acquisition for any issues related to SuperMicro, including re-examining a third-party security audit[5] that we conducted in 2015 as part of our due diligence prior to the acquisition. We’ve found no evidence to support claims of malicious chips or hardware modifications.[6]

[4] “our records” — whose records and what kind? Identity needs clarification as well as the type of records.

[5] Who is the third-party security auditor? How and why were they engaged?

[6] What about evidence of bad or mismatched firmware and firmware updates?

The pre-acquisition audit described four issues with a web application (not hardware or chips)[7] that SuperMicro provides for management of their motherboards. All these findings were fully addressed before we acquired Elemental. The first two issues, which the auditor[8] deemed as critical, related to a vulnerability in versions prior to 3.15 of this web application (our audit covered prior versions of Elemental appliances as well), and these vulnerabilities had been publicly disclosed by SuperMicro on 12/13/2013.[9]

[7] “web application” — but not firmware?

[8] Is this still the unnamed third-party security auditor or an internal auditor employed by Amazon or a subsidiary?

[9] How was this “publicly disclosed by SuperMicro”? SMCI’s website does not currently have either a press release or an SEC filing matching this date (see screenshots at bottom of this page).

Because Elemental appliances are not designed to be exposed to the public internet, our customers are protected against the vulnerability by default.[10] Nevertheless, the Elemental team had taken the extra action on or about 1/9/2014 to communicate with customers and provide instructions to download a new version of the web application from SuperMicro (and after 1/9/2014, all appliances shipped by Elemental had updated versions of the web application).[11] So, the two “critical” issues that the auditor found, were actually fixed long before we acquired Elemental. The remaining two non-critical issues with the web application were determined to be fully mitigated by the auditors if customers used the appliances as intended, without exposing them to the public internet.[12]

[10] “exposed to the public internet” — did customer data run through Elemental’s Supermicro devices between 2013 and 2015?

[11] What about firmware?

[12] Did customer data still run through devices with the two non-critical issues? Are any machines with these non-critical issues still in production?

Additionally, in June 2018, researchers made public reports of vulnerabilities in SuperMicro firmware.[13] As part of our standard operating procedure, we notified affected customers promptly, and recommended they upgrade the firmware in their appliances.[14]

[13] Researchers at Eclypsium are reported to have told Supermicro of vulnerabilities in January 2018. When was Amazon, AWS, or other Amazon subsidiary notified of these vulnerabilties?

[14] Give the six-month gap between Eclypsium’s notification to Supermicro and the public’s notification, when were Amazon’s, AWS’, or other Amazon subsidiary’s customers notified of these vulnerabilties?

__________

Screenshots

Supermicro’s SEC filings – last of year 2013:

Supermicro’s press releases – last of year 2013:

Rattled: China’s Hardware Hack – Apple’s Response

[NB: Note the byline. Portions of my content are speculative. / ~Rayne]

The following analysis includes a copy of an initial response received from Apple by Bloomberg Businessweek in response to its story, The Big Hack. In tandem with the Bloomberg story this was published on October 4 at this link. Apple’s response is offset in blockquote format. No signer was indicated in the published response. Additional responses from Apple to Bloomberg’s story will be assessed separately in a future post.

This analysis is a work in progress and subject to change.
__________

Apple

Over the course of the past year, Bloomberg has contacted us multiple times with claims, sometimes vague and sometimes elaborate, of an alleged security incident at Apple.[1] Each time, we have conducted rigorous internal investigations based on their inquiries and each time we have found absolutely no evidence to support any of them.[2] We have repeatedly and consistently offered factual responses, on the record, refuting virtually every aspect of Bloomberg’s story relating to Apple.[3]

[1] Phrasing avoids who made the allegation(s).

[2] “rigorous internal investigations” doesn’t describe what they actually investigated; “each time” refers to investigations AFTER Bloomberg contacted Apple, AFTER 2016 when Apple had broken off relations with Supermicro.

[3] “refuting virtually aspect” does not mean “every and all.”

On this we can be very clear: Apple has never found malicious chips, “hardware manipulations” or vulnerabilities purposely planted in any server.[4] Apple never had any contact with the FBI or any other agency about such an incident.[5] We are not aware of any investigation by the FBI, nor are our contacts in law enforcement.

[4] (a) What about problems with firmware updates, including malicious firmware, firmware not issued by Supermicro, or hijacking to firmware upgrade sites not created by Supermicro?

(b) “purposely planted in any server” refers not to Supermicro’s motherboards but Elemental or other server assemblies.

[5] What about contact with any government agency regarding firmware? What about contact with a third-party entity regarding firmware problems, including security researchers?

[6] This phrasing focuses on law enforcement but not on other possibilities like intelligence entities or non-law enforcement functions like Commerce or Treasury Departments.

In response to Bloomberg’s latest version of the narrative, we present the following facts: Siri and Topsy never shared servers;[7] Siri has never been deployed on servers sold to us by Super Micro; and Topsy data was limited to approximately 2,000 Super Micro servers, not 7,000. None of those servers has ever been found to hold malicious chips.[9]

[7] (a) What about earlier versions of Bloomberg’s narrative the public hasn’t seen?

(b) Did Siri and Topsy ever share a data farm facility?

[8] (a) Was Siri ever deployed on Elemental brand servers?

(b) Was Topsy ever deployed on Elemental brand servers?

[9] Did any of the servers on which Siri and Topsy were deployed experience firmware problems including malicious firmware, firmware not issued by Supermicro, or hijacking to firmware upgrade sites not created by Supermicro?

As a matter of practice, before servers are put into production at Apple they are inspected for security vulnerabilities and we update all firmware and software with the latest protections. We did not uncover any unusual vulnerabilities in the servers we purchased from Super Micro when we updated the firmware and software according to our standard procedures.[10]

[10] Is this a statement of current practices or practices during the period of time about which Bloomberg reported? Why did Apple end its relationship with Supermicro?

We are deeply disappointed that in their dealings with us, Bloomberg’s reporters have not been open to the possibility that they or their sources might be wrong or misinformed. Our best guess is that they are confusing their story with a previously-reported 2016 incident in which we discovered an infected driver on a single Super Micro server in one of our labs.[11] That one-time event was determined to be accidental and not a targeted attack against Apple.[12]

[11] Gaslighting about the journalists’ credibility. Have there ever been any servers from Elemental or other server manufacturer with “infected drivers,” including the “single Super Micro server in one of our labs”? Were any servers of any make with “infected drivers” in production environments, whether they faced customers or not?

[12] How is an “infected driver” an accident?

While there has been no claim that customer data was involved, we take these allegations seriously and we want users to know that we do everything possible to safeguard the personal information they entrust to us.[13] We also want them to know that what Bloomberg is reporting about Apple is inaccurate.[14]

[13] This is not the same as saying “customer data was not exposed.”

[14] “inaccurate” but not “wrong,” “erroneous,” “false,” or “untrue”?

Apple has always believed in being transparent about the ways we handle and protect data.[15] If there were ever such an event as Bloomberg News has claimed, we would be forthcoming about it and we would work closely with law enforcement.[16] Apple engineers conduct regular and rigorous security screenings to ensure that our systems are safe. We know that security is an endless race and that’s why we constantly fortify our systems against increasingly sophisticated hackers and cybercriminals who want to steal our data.[17]

[15] Tell us about iPhone encryption.

[16] “an event” is not “events”. “Forthcoming” may not mean “public disclosure” or “reveal that we are under non-disclosure agreements.” “Would work closely with law enforcement” is not the same as “working with intelligence community,” or “working with Commerce/Treasury Departments.”

[17] No specific mention of nation-state actors.

Rattled: China’s Hardware Hack

[NB: Note the byline. Portions of my analysis may be speculative. / ~Rayne]

As I noted in my last Three Things post, information security folks are rattled by the October 4 Bloomberg Businessweek report that extremely tiny microchips may have been covertly embedded in motherboards used by U.S. businesses.

Their cognitive dissonance runs in two general directions — the feasibility of implanting a chip at scale, and the ability of such a chip to provide a viable backdoor to a device.

Hardware security researchers and professionals have been debating manufacturing feasibility and chip ability across Twitter. Joe Fitz’ recent tweet threads suggest implantation of a rogue chip is entirely doable on a mechanical basis though what happens once a chip has been embedded must be assessed from a software perspective. Fitz is not alone in his assessment; other professionals and academics believe it’s possible to insert a ‘malicious’ chip. Computer security academic Nicholas Weaver pointed to small devices which could do exactly what the Bloomberg report suggested if these tiny objects were embedded into motherboards during manufacturing.

The feasibility also requires the right opportunity — a confluence of personnel, manufacturing capability and capacity, timing and traceability. Let’s say a rogue or compromised employee manages to slip chips into a batch of motherboards; which ones? To whom will they ship? How could a rogue/compromised employee ensure the motherboards left the facility undetected?

The Bloomberg report paints the U.S.-based Supermicro plant as a perfect environment in which such hardware infiltration could happen easily. With employees divided by two very different languages — English-speakers far less likely to understand Mandarin-speakers — discussions between multiple rogue/compromised employees could be very easy as would be sharing of written instructions. Supermicro’s ISO certifications for standards 9001, 13485, 14001, and 27001 may shed some light on how the company expected to manage two different languages in the same workplace.

One could argue a bilingual workplace shouldn’t pose a challenge given how many companies already use English/Spanish, English/French, or English/German. Compare, however, these words:

English: hardware

German: either hardware or computerhardware

French: either hardware or le matériel

Spanish: either hardware or los equipos

Mandarin: 硬件 (yìng jiàn)

With enough exposure the average English-as-primary-language worker could readily understand the most common western language words for equipment they were manufacturing. It would take considerably more investment in education to recognize and understand a pictographic language making casual quality control difficult.

The environment is even more challenging for mixed language staff in manufacturing plants located in China.

~ | ~ | ~

Let’s look at a timeline of events leading up to the Bloomberg report this week. Note how often the word ‘firmware‘ is used in this timeline and in the responses from Apple and Amazon to the Bloomberg story:

1993 — Charles Liang launched Supermicro.

2007 — Social search analytics company Topsy founded.

2005 — Defence Science Board warned “trojan horse” chips bought overseas could negatively affective military systems.

2008 — BusinessWeek reported that fake Chinese-made microchips had entered the military’s supply chain causing system crashes.

2010 — Defence Department bought 59,000 chips, unaware they were counterfeit.

2Q2011 — China denied entry visas to senators Levin and McCain staff for congressional probe in Guangdong province.

October 2011 — Apple releases Siri.

December 2013 — Apple acquired  Topsy.

December 2013 — Supermicro publicly disclosed vulnerability/ies in a web application related to management of motherboards (Amazon response, email Oct 2018)

December 2013 — CBS’ 60 Minutes program aired a story about the NSA in which a plot involving a rogue BIOS had been identified.

First half 2014 (date TBD) — Intelligence officials tell White House that PRC’s military would infiltrate Supermicro’s motherboard production with microchips intended for the U.S. market.

January 2014 — Elemental communicated to existing customers that a new version of the web app was available for download; equipment shipped after this date had updated versions of the web app. (Amazon response, email Oct 2018)

Early 2015 — Amazon launched pre-acquistion evaluation of startup Elemental Technologies which used Supermicro motherboards in servers it made.

Late spring 2015 — Elemental sent several servers to Ontario CAN for testing by third-party security firm. It found non-spec chips on server motherboards. (Bloomberg report)

May 2015 — Apple detected unusual network activity and experienced firmware problems.

Summer 2015 — Apple found non-spec chips on Supermicro motherboards Apple bought from Supermicro. (Bloomberg report)

September 2015 — Amazon announced its acquisition of Elemental.

December 2015 — Apple shut down Topsy.

Mid-2016 — Apple broke off its relationship with Supermicro.

June 2018 — Researchers publicized vulnerabilties found in Supermicro firmware. AWS notified customers and recommended a firmware upgrade. (Amazon response, email Oct 2018)

October 2018 — Amazon, Apple, Supermicro, and PRC submitted responses denying Bloomberg’s report. (Published by Bloomberg)

~ | ~ | ~

Follow up reporting by other news outlets increase the layers of denial that cloud companies Amazon and Apple were affected by a possible breach of the hardware supply chain.

Some have asked if Bloomberg’s report is merely an attempt to undermine Amazon and Apple, which are the two most valuable companies in the U.S. and in Apple’s case, the world.

It is their value and their place in the stock market along with the customers they serve which may drive some of the denial.

Remember that Amazon’s AWS has provided hosting to U.S. government agencies. Government employees also use Apple iPhones and by extension, Apple’s cloud services. Is it at all possible that in providing services to government agencies these corporations and/or their subsidiaries have been read into programs obligating a degree of secrecy which includes denial of vulnerabilities and breaches which do not affect directly the average non-governmental user of Amazon and Apple products and services?

~ | ~ | ~

There are additional events which appear to have happened independently of the alleged hardware supply chain infiltration. They may be extremely important and highly relevant if looked at from an industry and intelligence perspective.

March 2014Freescale Semiconductor lost 20 employees in apparent crash of Malaysia Air flight MH370 en route to Beijing. The employees were supposed to begin work on a new chip manufacturing facility in China. While Freescale’s chips were not those one might ordinarily associate with server motherboards, it’s worth asking if Freescale at that time had any chips which might have served as server chips, or if they could work as illicit hardware hacks when embedded in a motherboard. Freescale has since been acquired by NXP.

Late 2010 — Beginning in late 2010, China identified and executed a network of U.S. agents within its borders over a two-year period, resulting in the deaths of at least 30 persons and the prosecution of former CIA agent Jerry Chung Shin Lee who worked as an informant for PRC. The exposure of these spies was blamed in part on a compromised communications system which had been previously used in the middle east. Due to compartmentalization of the project, it’s reported Lee could not have identified the agents, placing more emphasis on the communications system.

Mid-2011 — China refused visas to staff for senators Carl Levin and John McCain for the purposes of investigating electronic components manufacturing in city of Shenzhen in Guangdong province. The congressional probe sought the source of counterfeit parts which had entered the U.S. military’s supply chain; U.S. Commerce Department reported in January 2010 that 400 companies surveyed “overwhelmingly cited China” as the point of origin for counterfeit parts.

These events spawn more questions when looking at technology supply chain hacking and communications systems which rely on this supply chain.

Did Freescale’s plans to expand production in China pose a risk to the hardware supply chain hack? Or was it simply a fluke that a substantive portion of the company’s manufacturing engineers disappeared on that flight? Though Freescale originated in Austin, Texas, it had a presence in China since 1992 with at least eight design labs and manufacturing facilities in China as of 2014.

Was the communications system used by doomed U.S. assets in China affected not by tradecraft or betrayal, or even by counterfeit parts, but by the hardware supply chain hack — and at an even earlier date than the timeline of events shown above related to Supermicro’s compromised motherboard production?

Did China refuse admittance to Guangdong province in 2011 related not to counterfeit parts but to the possibility that supply chain hacks beyond counterfeiting alone might be revealed?

Is the supply chain hack reported by Bloomberg part of a much larger security threat which has been slowly revealed but not widely acknowledged because the threat has been viewed through narrow military, or intelligence, or tech industry lenses?

The tech industry may be rattled by allegations that the computer hardware supply chain has been hacked. But the possibility this hack has gone on much longer and with massive potential collateral damage may truly shake them up.

~ | ~ | ~

There is a third train of cognitive dissonance, not limited to information security professionals. Persons outside the tech industry have indulged in denialism, taking comfort in the aggressive pushback by Apple and Amazon which each claim in their own way that the Bloomberg report is inaccurate. (I have an analysis of the early responses by Apple and Amazon; I will also examine later expanded responses as well as Supermicro’s and PRC’s responses as soon as time permits.)

But there have been reports for years about counterfeit electronic components, obstruction of investigations into these components, system failures which could be attributed to hardware or software which do not meet specifications. Cognitive dissonance also resists Bloomberg’s report that as many as 30 U.S. companies were affected, not just Apple and Amazon which have offered up high-profile rebuttals.

And there have been reports in industries outside of cloud services and the military where off specification or counterfeit electronic components have made it into production. One such anecdote appears in a thread at Hacker News YCombinator, discussing credit card payment systems and development of screening systems requiring application of tests using angular momentum to determine if a board has been altered without breaking the board’s tamper-proof seal.

In addition to his early tweets assessing feasibility of malicious or covert off-spec chips added to motherboards, Nicholas Weaver wrote a post for Lawfare about the Bloomberg report.

The Bloomberg story also explains a previous mystery: in 2016, Apple quietly removed all SuperMicro servers from their products due to an unspecified “Security Incident.”  At the time the rumor was that SuperMicro provided a sabotaged BIOS—that is, the bootstrap program used to start the computer, another “god mode” target for compromise. Apple denied then that there was any security incident—just as they are denying one now.

This incident once again illustrates the “Coventry problem,” referring to Winston Churchill’s apocryphal decision not to prevent the bombing of Coventry in order to keep secret that British intelligence had decrypted the Enigma machine. Robertson and Riley describe a U.S. intelligence apparatus that knew of these ongoing attacks, but could not effectively notify the affected companies nor provide useful recommendations. If the intelligence community had warned these companies, it would probably have revealed to the Chinese that the U.S. was aware of these activities, as well as potentially compromise an ongoing FBI investigation described in the article.

Weaver called the suspect Supermicro firmware a ‘BIOS’ — the first use of this term across multiple reports covering the Bloomberg report and its aftermath. This change in nomenclature is critical, particularly so given the point he makes about the “Coventry problem.” The term ‘BIOS’ does not appear in the early responses from Apple, Amazon, or Supermicro.

In December 2013, CBS’ 60 Minutes aired a report about the NSA; it appeared at the time to puff up the agency after the publication of Edward Snowden’s leaked documents about the government’s domestic spying using  PRISM. Within the story was a claim about a thwarted cyberattack:

Debora Plunkett: One of our analysts actually saw that the nation state had the intention to develop and to deliver, to actually use this capability— to destroy computers.

John Miller: To destroy computers.

Debora Plunkett: To destroy computers. So the BIOS is a basic input, output system. It’s, like, the foundational component firmware of a computer. You start your computer up. The BIOS kicks in. It activates hardware. It activates the operating system. It turns on the computer.

This is the BIOS system which starts most computers. The attack would have been disguised as a request for a software update. If the user agreed, the virus would’ve infected the computer.

John Miller: So, this basically would have gone into the system that starts up the computer, runs the systems, tells it what to do.

Debora Plunkett: That’s right.

John Miller: —and basically turned it into a cinderblock.

Debora Plunkett: A brick.

John Miller: And after that, there wouldn’t be much you could do with that computer.

The description sounds remarkably like the rogue firmware update in concert with a malicious/covert chip.

The manner in which this report was handled by the NSA, however, made it appear like disinformation. The assessment that such firmware would be used solely brick a device heightened the FUD around this report, deterring questions about applications other than bricking a device — like taking control of the computer, or collecting all its transaction and data. Was the FUD-enhanced release via 60 Minutes the intelligence community’s approach to the “Coventry problem”?

~ | ~ | ~

The problem Bloomberg’s Jordan Robertson and Michael Riley reported is probably much bigger than they described. It is bigger than Supermicro motherboards and firmware, and it’s not a problem of the near-term future but ongoing over the last decade.

At what point will U.S. industries organize a collective response to both counterfeit and off-specification manufacturing of electronic components overseas? They can’t count on a calm and rational response from the Trump administration given the unnecessary trade war it launched against China.
_____

Disclosure: I have positions in AAPL and AMZN in my investment portfolio.

Three Things: Russia and China Spying, Kavanope

[NB: Yes, it’s Rayne, not Marcy. Check the byline.]

Huge news earlier today related to spying. Really big. MASSIVE.

And a MASSIVE cover-up pawned off on the feeble-minded as a ‘complete investigation‘ into Dr. Ford’s and Deborah Ramirez’s accusations against Brett Kavanaugh.

~ 3 ~

Bloomberg published an epic piece of investigative journalism this morning about China’s spying on U.S. businesses by way of tiny chips embedded in server motherboards. The photos in the story are just as important as the must-read story itself as they crystallize a challenge for U.S. intelligence and tech communities. Like this pic:

That tiny pale obelisk to the right of the penny represents one of the malicious chips found in affected Supermicro brand motherboards shipped to the U.S. market — nearly as small as the numbers in the date on the coin. Imagine looking for something this puny before a machine is turned on and begins to launch its operating system. Imagine trying to find it when it is sandwiched inside the board itself, embedded in the fiberglass on top of which components are cemented.

The chip could undermine encryption and passwords, making any system open to those who know about its presence. According to Bloomberg reporters  Jordan Robertson and Michael Riley, the chips found their way into motherboards used by Apple and Amazon.

Information security folks are scrambling right now because this report rocks their assumptions about the supply chain and their overall infosec worldview. Quite a few doubt this Bloomberg report, their skepticism heightened by the carefully worded denials offered by affected and relevant parties Apple, Amazon, Supermicro, and China. Apple provided an itemization of what it believed Bloomberg Businessweek got wrong along with its denial.

I’ll have more on this in a future post. Yes, indeedy.

~ 2 ~

A cooperative, organized response by Britain, The Netherlands, U.S., and Canada today included the indictment of seven Russians by the U.S. for conspiracy, conspiracy to commit wire fraud, wire fraud, aggravated identity theft, and conspiracy to launder money. The Russians have been identified as members of a GRU team organized out of a facility in Moscow, working on hacking and a disinformation influence campaign focused on anti-doping entities and non-Russian Olympic athletic competitors.

Note the underlined bit in this excerpt from the indictment (pdf) — the last indictment I copied with similar wording was that of Evgeny Buryakov and his two comrades, the three spies based in New York City who worked with “Male-1”, now known to be Carter Page. Who are the known and unknown? Persons who have flipped or co-conspirators yet to be named?

The UK released a statement as did the Canadians, and Netherlands issued a joint statement with the UK about the entirety of spying for which this GRU team is believed to be responsible, including an attempt to breach the Organisation for the Prohibition of Chemical Weapons’ (OPCW) facility analyzing the Novichok nerve agent used to poison the Skripals in the UK as well as chemicals used against Syrians.

Cryptocurrency news outlets report concerns that this indictment reveals the extent of USDOJ’s ability to trace cryptocurrency.

An interesting coincidence took place overnight as well — Russian Deputy Attorney General Saak Karapetyan died last night when an unauthorized helicopter flight crashed northeast of Moscow. Karapetyan had been linked this past January to Natalia Veselnitskaya and an attempt to recruit Switzerland’s top investigator as double-agents. But Karapetyan had also been involved in Russia’s response to the poisoning of Alexander Litvinenko and the aftermath of the Skripals’ poisoning in the UK.

What remarkable timing.

One might wonder if this accident had anything to do with the unusual release of GRU personnel details by the Dutch Military Intelligence and Security Service (MIVD) and the United Kingdom’s Ministry of Justice during their joint statement today.

By comparing the released identity documents, passports, automobile registrations and the address provided when cars were rented, the identities of a total 305 GRU agents may have been identified by bellingcat and The Insider including the four out of the seven men wanted by the U.S. for the anti-doping hackingas well as attempted breach of OPCW.

The identity of the four GRU agents accused of targeting the OPCW was cinched by a taxi receipt in one agent’s pocket from a location on the road next to the GRU’s facility in Russia. Four agents also had consecutive passport numbers.

What remarkably bad opsec.

~ 1 ~

As for the impending vote on Brett Kavanaugh:

– Senator Heidi Heitkamp is voting her conscience — NO on Kavanaugh.
– Senator Joe Manchin is now the lone Dem holdout; he says he’s still listening but hasn’t seen anything incriminating from Kavanaugh’s adulthood. (Gee, I wonder why.)
– Senator Bob Menendez didn’t mince words. He said “It’s a bullshit investigation.” (He should know what a thorough investigation looks like).

And the beer-loving former Yale frat boy had an op-ed published in the Wall Street Journal which pleads with us to lose all intelligence and believe that he is really very neutral. I am not even going to link to that POS which has re-enraged women all over the country.

GTFO.

Continue calling your senators to thank them for a NO vote on Kavanaugh so that they aren’t hearing right-wing demands alone. Congressional switchboard: (202) 224-3121

~ 0 ~

This is an open thread. Sic ’em.

Three Things: CRC—What? An Indictment, Plus Shut Downs Ahead

[NB: As always, check the byline. / ~Rayne]

Brett Kavanaugh’s nomination and confirmation process is an 800-pound gorilla in the media, as is the potential for the obstructive removal of Rod Rosenstein as Deputy Attorney General. They suck up enormous amounts of mental wattage, sitting wherever they want to sit.

Here are three things which are in some way related and worth more of our attention, whatever is left after the gorillas are done with it.

~ 3 ~

CRC: One degree from Manafort

Thomas Fine went prowling around FARA filings, landing this juicy find (pdf):

Yes, Creative Response Concepts, Inc., the same firm for which Ed Whelan has worked, registered in 2005 as a foreign agent for Viktor Yanukovych — the same Yanukovych for which Paul Manafort also worked as an illegal foreign agent. CRC was paid $10,000 by Potomac Communications Group, for which Aleksei Kiselev worked. Kiselev also worked for Paul Manafort to assist Yanukovych.

What a small, small world.

Should note CRC’s registration was after the fact — they were contracted for April-October 2003. Why so late?

(Thanks to @JamesFourM for the PCG-Kiselev-Manafort link.)

~ 2 ~

Indictment yesterday related to Trump Towers…in Azerbaijan

Didn’t see this until late last night: DOJ indicted Kemal “Kevin” Oksuz (pdf) on one count of hiding or falsifying material facts and four counts of making false statements to the U.S. House of Representatives Committee on Ethics. The filings were related to a Congressional trip to Azerbaijan ultimately paid for by State Oil Company of Azerbaijan Republic (SOCAR), the wholly state-owned national oil and gas company of Azerbaijan.

Oksuz is now a fugitive.

Ten members of Congress and 32 staffers traveled in 2013 to attend a U.S.-Azerbaijan convention in Baku after Azerbaijan had asked Congress for an exemption from sanctions on Iran for a $28 billion natural gas pipeline project. The members and staffers were later cleared as it appeared they believed the trip’s funding was provided by Oksuz’s nonprofit organization.

Personally, I think those members and staffers needed a rebuke. Nonprofits don’t print money; they rely on money from donors. Follow the money to the donors before accepting a trip and incidentals. It’s not rocket science.

Worth keeping in mind the Trump International Hotel & Tower built in Baku, overseen by Ivanka Trump, which burned in late April this year — an amazing two fires, same day. What are the odds?

~ 1 ~

Shutdowns Ahead: U.S.-Canada and U.S. Government?

Doesn’t look like negotiations between the U.S. and Canada are going to make this Saturday’s deadline. No idea what will happen after that. We all know the Trump administration has been at fault; how could anybody screw up a long-term peaceful relationship like U.S.-Canada, our second largest trading partner after China, without deliberate bad faith? Without the intent to screw over another NATO member’s economy?

And the U.S. government itself faces a budget deadline. If the “minibus” budget bill isn’t signed by midnight this coming Sunday we’re looking at a shutdown and it appears the bottleneck may be Trump. The jerks at Breitbart are fomenting to encourage a shutdown by insisting Trump refuse to sign the bill — they’re just plain malicious, thinking not at all about the impact on fellow Americans or the economy.

Putin must be laughing his ass off at how easily the GOP’s white nationalist base has subverted U.S. and NATO stability by giving up control to a mobbed-up, golf-addicted, attention-deficient wig.

~ 0 ~

Don’t miss Marcy’s interview on Democracy Now in which she talks about Rod Rosenstein’s status and the Kavanaugh confirmation process.

Treat this like an open thread — have at it.

p.s. A note on site operations: Please be sure to use the same username and email address each time you log into the site. It makes it easier for community members to get to know you. Deliberate sockpuppeting is not permitted.

Legal Ethics in Trumplandia

Warning: this post may be considered uncivil.

I was a public servant for 6 ½ years. I was an assistant in the Tennessee State Attorney General’s office beginning in mid-1977, and became Securities Commissioner in the Insurance Department in mid-1980. In that time I dealt with a number of interesting ethical issues, directly and indirectly. Where do we draw the line between defending the constitutionality of a questionable statute? Should we intervene in a specific case for public policy reasons? Should we defend a lawsuit against a state employee?

Particularly difficult questions arose when suits were filed alleging systematic violations of law or human rights. The Alcoholic Beverage Commission was being bribed into approving liquor licenses, for example. A worse case involved guards sexually assaulting juvenile detainees. I won’t discuss these cases even now, but I learned the importance of making decisions as a lawyer that kept my conscience clear.

It must be like this every half-hour nowadays for career Department of Justice lawyers. The ignorant policy decisions, the incompetent drafting, the table-pounding public statements on top of difficult questions of constitutional law and statutory interpretation must make for situations that are fascinating, difficult, and even funny in a bizarre way. I don’t have a problem with career lawyers defending the policies of this or any administration. I do question some of their arguments. For example, in the Muslim Ban cases they argued that public statements made as candidate and as President aren’t relevant, which seems ridiculous, but SCOTUS disagrees with me so I was wrong. Or something was wrong. But anyway, I know this must be exhilarating for those people, and I hope they are finding the pleasure that I can see from a distance in the kinds of issues they face.

That doesn’t apply to the Child Snatching Case. Or, as the normalizing media call it, the Child Separation Policy. The facts of the matter are not seriously disputed. As a matter of policy, every person deemed to have entered the US illegally is charged with a crime. That includes people lawfully seeking asylum. Their children are snatched from their arms and sent thousands of miles away. The parents are jailed. The kids are kept in cages before transfer, often to horrifying profit-making entities where care is minimal. Some of the kids are drugged without their consent or that of their parents. The government doesn’t know where the kids are or how to reunite them. Some of the parents were deported without their children. The policy of referring all immigrants for prosecution may have been dropped recently.

The policy was put into effect secretly, with no notice, in the Summer of 2017, and the government formally admitted it April 6, 2018. At least 2,000 thousand children were snatched. Here’s a short history from the New Yorker.

The ACLU filed suit February 26, 2018 on behalf of a Congolese woman who sought asylum for herself and her 7 year old daughter who were separated pursuant to the policy. DOJ lawyers entered an appearance March 23, and filed a motion to dismiss April 6. The ACLU filed an amended complaint, and then a request for a preliminary injunction. The DOJ lawyers objected. A hearing was held in June, and a preliminary injunction entered June 26. It became clear at that point that the Trump Administration couldn’t find kids, didn’t know which kid went with which parent, and didn’t have any way to find the parents who had been deported without their children.

Buzzfeed has done a good job reporting on this case. Here’s a report by Adolfo Flores on the July 7 hearing that clarified the sickening state of the records and the failure of the Trump Administration to protect the children. Here’s Zoe Tillman’s report on the status as of July 9. Apparently one family that was separated were US citizens.

This policy punishes parents, many of whom are innocent, without due process. All of the children are damaged, and they are all innocent of any wrong-doing. The punishment is cruel and unusual in the sense normal people use those words.

The policy, if this unplanned and undocumented perversion can be called a policy, was imposed by US Attorney General Jeff Sessions. He says it was designed to deter families from illegal entry, but that is an easily disproved lie. It couldn’t work if it wasn’t public. It couldn’t work if entry is legal, as in the case of those seeking asylum. And it could never work against the children. This people who designed and approved this policy are sadists. They’re just the latest version of US monsters, like the torturers, the liars who ginned up the Iraq War, and the armchair warriors who send out the bombers and cruise missiles as the mood or politics strikes them. We have no recourse against them. They are beyond the reach of law or conscience. They are beyond accountability.

The people who are carrying out this policy are the only people who could have ended it. They didn’t. They are complicit and each one bears a share of guilt.

That includes the lawyers who defended the case. Assuming a minimum degree of competence, I speculate that the DOJ lawyers in the ACLU case knew about the policy and had some idea of the scope of the damage by mid-March. They certainly knew about the policy and its purpose by the date of the public announcement, April 6, the day they filed the motion to dismiss. They then chose to continue to litigate rather than work to terminate of this inhuman policy and stop the damage, and failing that, to resign noisily, They had choices; hard choices to be sure because the policy was designed by their political boss, Jeff Sessions, but still choices.

The effect of their decision to continue litigating is that the life of the policy was prolonged for months and more children were snatched. Other workers were put in a position where they may have felt they had no choice but to enforce the policy. The government and its private contractors continued to abuse the parents and especially the children. The resources of charitable organizations and others working on this disaster were depleted. Surely the lawyers didn’t need the money or the job that badly.

They may still have a conscience. If so, I hope it eats at them all their lives. I hope they have to explain their actions to their children. I hope the memory of toddlers screaming for their moms and dads comes to them in the night and gives them sweats. It’s a fair price to pay for the damage they have done to thousands of children and their parents.

image_print