Cloud Computing and the Single Server

[NB: Check the byline, thanks. /~Rayne]

I’ve been meaning to write about this for a while. Push came to shove with Marcy’s post this past week on Roger Stone and the Russian hack of the DNC’s emails as well as her post on Rick Gates’ status update which intersects wih Roger Stone’s case.

First, an abbreviated primer about cloud computing. You’ve likely heard the term before even if you’re not an information technology professional because many of the services you use on the internet rely on cloud computing.

Blogging, for example, wouldn’t have taken off and become popular if it wasn’t for the concept of software and content storage hosted somewhere in a data center. The first blogging application I used required users to download the application and then transfer their blogpost using FTP (file transfer protocol) to a server. What a nuisance. Once platforms like Blogger provided a user application accessible by a browser as well as the blog application and hosting on a remote server, blogging exploded. This is just one example of cloud computing made commonplace.

Email is another example of cloud computing you probably don’t even think about, though some users still do use a local email client application like Microsoft’s proprietary application Outlook or Mozilla’s open source application Thunderbird. Even these client applications at a user’s fingertips rely on files received, sent, managed, and stored by software in a data center.

I won’t get into more technical terms like network attached storage or storage area network or other more challenging topics like virtualization. What the average American needs to know is that a lot of computing they come in contact every day isn’t done on desktop or laptop computers, or even servers located in a small business’s office.

A massive amount of computing and the related storage operates and resides in the cloud — a cutesy name for a remotely located data center.

This is a data center:

Located in Council Bluffs, Iowa, this is one of Google’s many data centers. In this photo you can see racks of servers and all the infrastructure supporting the servers, though some of it isn’t readily visible to the untrained eye.

This is another data center:

This is an Amazon data center, possibly one supporting Amazon Web Services (AWS), one of the biggest cloud service providers. Many of the sites you visit on the internet every day purchase their hosting and other services from AWS. Some companies ‘rent’ hosting space for their email service from AWS.

Here’s a snapshot of a technician working in a Google data center:

Beneath those white tiles making up the ‘floor’ are miles and miles of network cables and wiring for power as well as ventilation systems. More cables, wires, and ventilation run overhead.

Note the red bubble I’ve added to the photo — that’s a single blade-type server inserted into a rack. It’s hard to say how much computing power and storage that one blade might have had on it because that information would have been (and remains) proprietary — made to AWS specifications, which change with technology’s improvements.

These blades are swapped out on a regular maintenance cycle, too, their load shifted to other blades as they are taken down and replaced with a new blade.

Now ask yourself which of these servers in this or some other data center might have hosted John Podesta’s emails, or those of 300 other people linked to the Clinton campaign and the Democratic Party targeted by Russia in the same March 2016 bulk phishing attack?

Not a single one of them — probably many of them.

And the data and applications may not stay in one server, one rack, one site alone. It could be spread all over depending on what’s most efficient and available at any time, and the architecture of failover redundancy.

~ ~ ~
Some enterprises may not rely on software-as-a-service (SaaS), like email, hosted in a massive data center cloud. They might instead operate their own email server farm. Depending on the size of the organization, this can be a server that looks not unlike a desktop computer, or it can be a server farm in a small data center.

(The Fortune 100 company for which I once worked had multiple data centers located globally, as well as smaller server clusters located on site for specialized needs, ex. a cluster collecting real-time telemetry from customers. Their very specific needs as well as the realistic possibility that smaller businesses could be spun off required more flexibility than purchasing hosted services could provide at the time.)

And some enterprises may rely on a mix of cloud-based SaaS and self-maintained and -hosted applications.

In 2016 the DNC used Microsoft Exchange Server software for its email across different servers. Like the much larger Google-hosted Gmail service, users accessed their mail through browsers or client applications on their devices. The diagrams reflecting these two different email systems aren’t very different.

This is a representation of Google’s Gmail:

[source: MakeInJava(.)com]

This is a representation of Microsoft Exchange Server:

Users, through client/browser applications, access their email on a remote server via the internet. Same-same in general terms, except for scale and location.

If you’ve been following along with the Trump-Russia investigation, you know that there’s been considerable whining on the part of the pro-Trump faction about the DNC’s email server. They question why a victim of a hack would not have turned over their server to the FBI for forensic investigation and instead went to a well-known cybersecurity firm, Crowdstrike, to both stop the hack, remove whatever invasive tools had been used, and determine the entity/ies behind the hack.

A number of articles have been written explaining the hacking scenario and laying out a timeline. A couple pieces in particular noted that turning over the server to the FBI would have been disruptive — see Kevin Poulsen in The Daily Beast last July, quoting former FBI cybercrime agent James Harris:

“In most cases you don’t even ask, you just assume you’re going to make forensic copies…For example when the Google breach happened back in 2009, agents were sent out with express instructions that you image what they allow you to image, because they’re the victim, you don’t have a search warrant, and you don’t want to disrupt their business.”

Poulsen also quantified the affected computing equipment as “140 servers, most of them cloud-based” meaning some email and other communications services may have been hosted outside the DNC’s site. It would make sense to use contracted cloud computing based on the ability to serve widespread locations and scale up as the election season crunched on.

But what’s disturbing about the demands for the server — implying the DNC’s email was located on a single computer within DNC’s physical control — is not just ignorance about cloud computing and how it works.

It’s that demands for the DNC to turn over their single server went all the way to the top of the Republican Party when Trump himself complained — from Helsinki, under Putin’s watchful eye — about the DNC’s server:

“You have groups that are wondering why the FBI never took the server. Why didn’t they take the server? Where is the server, I want to know, and what is the server saying?”

And the rest of the right-wing Trumpist ecosphere picked up the refrain and maintains it to this day.

Except none of them are demanding Google turn over the original Gmail servers through which John Podesta was hacked and hundreds of contacts phished.

And none of the demands are expressly about AWS servers used to host some of DNC’s email, communications, and data.

The demands are focused on some indeterminate yet singular server belonging to or used by the DNC.

~ ~ ~
The DNC had to shut down their affected equipment and remove it from their network in order to clean out the intrusion; some of their equipment had to be stripped down to “bare metal,” meaning all software and data on affected systems were removed before they were rebuilt or replaced. 180 desktops and laptops had to be replaced — a measure which in enterprise settings is highly disruptive.

Imagine, too, how sensitive DNC staff were going forward about sharing materials freely within their organization, not knowing whether someone might slip and fall prey to spearphishing. There must have been communications and impromptu retraining about information security after the hack was discovered and the network remediated.

All of this done smack in the middle of the 2016 election season — the most important days of the entire four-year-long election cycle — leading into the Democratic Party’s convention.

(This remediation still wasn’t enough because the Russians remained in the machines into October 2016.)

If the right-wing monkey horde cares only about the DNC’s “the server” and not the Google Gmail servers accessed in March 2016 or the AWS servers accessed April through October 2016, this should tell you their true aim: It’s to disrupt and shut down the DNC again.

The interference with the 2016 election wasn’t just Russian-aided disinformation attacking Hillary Clinton and allies, or Russian hacks stealing emails and other files in order to leak them through Wikileaks.

The interference included forcing the DNC to shut down and/or reroute parts of its operation:

(excerpt, p. 22, DNC lawsuit against Russian Federation, GRU, et al)

And the attack continues unabated, going into the 2020 general election season as long as the right-wing Trumpists continue to demand the DNC turn over the server.

There is no one server. The DNC shouldn’t slow or halt its operations to accommodate opponents’ and suspects’ bad faith.

~ ~ ~
As for Trump’s complaint from Helsinki: he knows diddly-squat about technology. It’s not surprising his comments reflected this.

But he made these comments in Helsinki, after meeting with Putin. Was he repeating part of what he had been told, that Russia didn’t hack the server? Was he not only parroting Putin’s denial but attempting to obstruct justice by interfering in the investigation by insisting the server needed to be physically seized for forensic inspection?

~ ~ ~
With regard to Roger Stone’s claims about Crowdstrike, his complaints aren’t just a means to distract and redirect from his personal exposure. They provide another means to disrupt the DNC’s normal business going forward.

The demands are also a means to verify what exactly the Special Counsel’s Office and Crowdstrike found in order to determine what will be more effective next time.

The interference continues under our noses.

This is an open thread.

image_print
63 replies
  1. Jockobadger says:

    Rayne, Thanks much for doing this primer. I cannot believe these people are such morons. I wonder how many of them actually believe that the “server” exists/existed and how many are simply using the whole notion as a tool to manipulate others. I know tr*mp isn’t that smart, but some of those in his orbit likely are. I pray this is over soon. JHC.

    • Rapier says:

      Your assuming they don’t know it’s BS. Why would you? It’s funny after all these years so many peoples first assumption that they, the GOP or really any politician, is acting in good faith.

      • Rayne says:

        Some of them know the demands are bullshit. But the base does not. And it’s not just Trump’s base — why do you think I put up a primer on cloud computing? The average American really doesn’t understand what’s going on behind their Google search or their shopping on Amazon. Our education system *still* doesn’t prioritize computer science even though nearly every aspect of our lives is shaped in some way by computing and increasingly by cloud computing.

        • P J Evans says:

          It’s a lovely clear explanation.
          (I remember one fire drill, we ended up standing next to a server room which had a window on the corridor where we were standing, so we could see the racks and the blades. Not a large room, either. The AC was high enough that the corridor was cold. And then there was the year we were temporarily on a floor with a “local” server right next to our area, where the AC went bad in that room and they brought in a portable unit to keep the local server happy; the other part of being on that floor was the opposite corner, where the meeting room was, was quite cold.)

          • Rayne says:

            LOL I always made sure I wore pants and took a sweater when I had to work in the data center. Crispy cool in there all the time. Nobody minded pulling a shift in there during the worst part of July-August dog days. Though the white noise I never really got use to — amazing how loud all the computers and ventilation systems can be.

        • KAL says:

          Fascinating and educational article. I am one of those Americans who had no clue until I read this. Thanks for the tutorial.

  2. P J Evans says:

    The physical servers for the stuff I worked with were in San Diego county, in one of the suburbs of San Diego. I worked in downtown L.A. It got interesting when the area where the physical servers were located was threatened by a major fire. (I hope they’ve thought about that situation and what to do when it happens again. Or if one of the faults in that part of the state decides to break.)

    • Rayne says:

      Most businesses relying on contract cloud computing using SaaS or PaaS (Platform as a Service) are also relying on redundancy for business continuity. The firm I worked at had failover to data centers 1500 and 3000 miles away. That said, every business buying cloud computing should ask about failover. It’s just best practice.

  3. Rapier says:

    Back in the day when the whole West Wing of the Bush White House was using an RNC Email server was that cloud based too or was it a system that was physically under the RNC’s control.

    Neither here nor there but it would help explain how they ‘lost’ 5 million emails. Need I mention they were supposed to use a government network, by law, didn’t, and it was decided no harm no foul or something like that.

    • timbo says:

      It was a scandal that the DP failed to fully follow up on too. But that’s water under the bridge now, isn’t it?

      • P J Evans says:

        And they were supposed to do what else, because they did everything they were supposed to do. Law enforcement does investigating and charging.

    • rip says:

      I’ve always suggested that our National Secret Archives (NSA – otherwise known as the National Security Agency or No Such Agency) should be used as the nation’s backup.

      If you managed to lose 50,000 emails between the VP and the energy companies – no problem – ask the NSA for copies (bulk discounts available.)

      It would amaze me if all communications, even made through privacy-oriented devices, aren’t archived.

      I use a VPN and other encrypting technologies for personal and business communications. I will assume I’m not totally private.

  4. viget says:

    Jeez… it’s like no one out there’s ever watched Mr. Robot…..

    But seriously, Rayne, bravo! This is the most succinct and informative takedown of the single server fallacy I’ve ever seen. And then there’s the fact that it’s not just emails that were hacked, it was polling data and internal research as well, hosted on AWS, completely separate from the emails.

    Thank you for all that you do, I would really love to buy you a drink and pick your brain some time!

    • Rayne says:

      Thanks. I didn’t want to address the other content hacked – it even included phone systems.

      I think Kevin Poulsen’s piece at The Daily Beast did a decent job, but he tends to write assuming an audience with some technical knowledge. As someone who worked both sides of the desk — the business person who bought/managed IT and the IT service provider — I often had to explain to techno-n00bs like senior management in the simplest terms what IT needed to do during technology repairs/upgrades. And I had to explain to technicians what the business’s concerns were with system changes. It’s challenging to explain to a VP why they can’t print a quarterly report when the cloud is down for quarter end and a long-scheduled upgrade across the network (“But the printer is right here…” Uh, it doesn’t work that way, sir. That’s why we told you 6 months/3 months/last month/this week/yesterday to get your prints ahead of downtime.) Sometimes one literally needs to draw a picture to explain things.

      You know what really irritates the fuck out of me even more knowing all this about the hack? “She was a bad candidate, didn’t visit key states late in the race…” How much was Clinton’s schedule late in the race affected not just by the disruption to the DNC and campaign team, possible tampering with data, advance notice to disinformation team to sow discord in social media affecting local polling, but the ongoing hack well into October?

      • timbo says:

        Yes, this. The DP didn’t seem to take this very seriously until the Mueller report was out. It boggles the mind why they weren’t talking about impeachment much earlier and now they’re being dragged kicking and screaming to the impeachment table. The DP in 1972/73 clearly got the threat they were facing—these folks? Not so much.

    • rip says:

      Trick question? Perhaps in one of those rooms like the NSA maintained inside AT&T at San Francisco? Like millions of other junction points that have a few extra wires leading out?

    • Rayne says:

      The series of tubes are under the “floor,” Senator Stevens. But seriously, the network runs for most data centers are under what look like floor tiles as well as cable trays overhead.

      Ignoring its rah-rah-go-Googleness, I think this video does a decent job sharing the operations of a data center, including all the “pipes” and “tubes.”
      https://m.youtube.com/watch?v=XZmGGAbHqa0

      (Surprise: there really are tubes — for the cooling system.)

  5. drouse says:

    There is still a substantial portion of the right that insists that the data was exfiltrated on a thumb drive. Like it was some Mission Impossible movie where you just stick the drive in a USB port and viola, the data just starts to download. Your average staffer isn’t going to have the permissions to do something like that or, for that matter, the skills to elevate their access to do it. The turn over the server ploy ignores the fact that a lot of hacking tools are memory resident and if you turn off the server the evidence go poof. Thus forensic images. As an aside, It seems that almost every day you see a story about someone failing to secure their cloud storage, exposing all kinds of data to whomever stumbles across it.

    • P J Evans says:

      They also don’t seem to be aware that CD/DVD drives and USB ports can be disabled, so they can’t be used for either copying or loading software or data. (It’s a nuisance when the legal software has to be updated.)

    • Rayne says:

      I neglected to mention RAM-resident malware – but try telling Google or AWS they need to shut off a blade server, RAM-resident malware or no.

      And quite honestly, cutting power to a device with RAM-resident malware would further the obstruction by cleaning up any traces not already removed by CCleaner or other methods.

  6. BobCon says:

    Another way to approach this is to think about the way large, old organizations like the DNC are operated. They are almost never run as a single integrated unit with a top down command and control structure. They consist of multiple units, often in some kind of conflict, with overlapping areas of authority, spread out over different geographic areas.

    The result is almost inevitably a fractured IT operation, with a division of email systems as part of that fracture. Pretty much every organization of this size and age has gone through multiple cycles where a central office IT exec has beaten the drum to centralize IT operations, resulting in a contraction of email systems, followed by an effort by subunits to break apart IT operations, resulting in a division of email systems. New leadership shows up with allegiances to a different set of IT consultants, regional officials have their own people they want to back, and priorities shift and change.

    You can pretty much guarantee that every organization like the DNC is at the beginning or middle (but never the end) of a plan to reorganize its email systems, and the actual hardware is in a state of permanent flux.

    • P J Evans says:

      The one I worked at had IT in charge of most of the software and hardware, and OpsTech in charge of the stuff used in my department (we had CAD and GIS software, which IT is not expert at handling). It worked most of the time…. (OpsTech was part of our department. They did some very specialized software.)

    • Rayne says:

      Rather describes the Fortune 100 company for which I worked. Largest, oldest sites would set the enterprise standards but smaller satellite locations might take years to catch up due to widely varying business models, budgeting, and availability.

      I won’t forget the project I managed integrating a newly-acquired smallish corporation with two plant sites. When we asked during email integration what their email servers’ names were the site staff told us, “Billie Joe and Bobbie Sue.”

      • Froggacuda says:

        Cute names still abound. The critical pair of firewalls for a major US air transport hub are literally named Boris and Natasha. I have a dozen other examples of current, in-production geek humor that I can’t mention, but it is safe to say that nerd nomenclature is alive and well.

        Earlier in my career, the solution to deploying new computers with USB ports in sensitive locations was not just to disable the ports in BIOs and OS, but to pack them full of super glue for good measure.

        • Rayne says:

          I think in this case it wasn’t nerd culture as much as Hicksville USA. If I told you where this business was located you’d slap your forehead and say, “Of course.” I swear half the town worked for the one site where the email servers were located. I pitied the integration engineers I had to send for the cutover.

        • J R in WV says:

          Boris and Natasha are great. Cute names are everywhere, still today.

          Our home network is based in my neighbor’s house, where the sat dishes are co-located. His wi-fi link is named “FBI Remote Analysis Van CF346”! He doesn’t think a hard password is necessary. I think he’s right!!

  7. viget says:

    Argh… Trump just pardoned Conrad Black, he of Hollinger International fame. Also known as one of another Pat Fitzgerald specials… Fitz must be livid.

    Although Fitz did make the mistake of doing the RICO there, which Black was ultimately acquitted of.

    Still… this is utter contempt for rule of law.

    • Tom says:

      At least Black’s time behind bars turned him into a spokesman for prison reform in the U.S. and Canada. In a May 12, 2012 story in “The Star” by reporter Alyshah Hasham, he is quoted as follows: “I would not meet the usual definition of a socialist … but many [of his fellow inmates] are victims of legal and social injustice, inadequately provided for by the social assistance system, and over-prosecuted and vengefully sentenced. The greater competitiveness of the world makes the failures of American education, social services and justice unaffordable as well as repulsive.”

  8. Marinela says:

    Regarding:
    The demands are also a means to verify what exactly the Special Counsel’s Office and Crowdstrike found in order to determine what will be more effective next time.

    Exactly. You nailed it.
    Great write up Rayne.

    So Trump wants “the DNC server” to the FBI, so that his FBI spies can inform his best body Putin how to do better in future hacks.

    The times we leave …

  9. roberts robot double says:

    As a long-time software systems architect and engineer/programmer, I appreciate how the horrors of having to communicate tech systems info to the pointy-haired-bosses of the world has given you the ability to craft such a nice write-up. Well done, indeed.

    One thing that didn’t convey — and may just be my lack of understanding of the facts — is how does Hillary’s “bathroom MS Exchange server” factor into the quotes from Trump et al? I mean, I never read anything about the DNC’s technical architecture regarding the Podesta email hacks and I’m sure you are spot-on about their being cloud-based, but as someone who fully understands the absolute horrors Exchange email servers present to a security-conscious architect (Exchange being to my knowledge Microsoft’s most egregious and long-lasting security nightmare in a sea of such products), are not a great deal of Trump’s “but her server” references targeted at that “bathroom server”, or do I have my facts wrong here?

    And, to be sure, I despise the RNC and especially Trump for their litany of blatant corruptions and other bald-faced bullshit, but Hillary having a Goddamned Exchange server in her bathroom was nothing less than batshit insane from a security standpoint and always gave me a very, very shady vibe. Absolutely, I would give one of my arms to install Ms. Clinton into the presidency, but whatever she was doing with that server and the fact that the FBI utterly destroyed it is sketchy a f.

    What I’m saying is, “Wasn’t that a case (no pun intended) of a single physical server (i.e. not cloud-based) that was not fully forensically analyzed?” Or do I have my facts wrong here?

    And you made a fine point I had never considered before: that the Russian fuckery likely very adversely affected the DNC’s ability to effectively run her campaign. That said, as a lurker on Schneier on Security for many, many years now, the voting machines are my primary worry, especially considering how (intentionally?) insecure they have been since Day #1, when they were Diebold. As such, wasn’t an election tampering report supposed to have been delivered to congress, and has been held up numerous times? I don’t remember that quarrel getting resolved — something about (per usual) the necessary agency ignoring a couple of 90-day (?) deadlines or somesuch.

    It is my cynical contention that the actual voting machines (and/or the servers that aggregate their data) were compromised and that the US intelligence community will *NEVER* cop to the level of actual statistical manipulation that potentially/likely occurred in 2016. I mean, for one, Trump would likely stroke-out if such info got released, but I’m mainly just referring to our gov’t’s not ever being willing to admit how badly it was pwned. I know it’s a completely separate dimension of escapade, but look how long it took for them to admit the Gulf of Tonkin incident was created from whole cloth.

    And for the non-technical out there, the most insidious and bleeding edge malware is the kind that infects the hardware subsystems that pervade modern hardware. Such hardware-based malware (e.g. BIOS and other device firmware as well as System Management Mode drivers) are completely independent of operating system and will survive wiping the hard drives. The sub-systems that facilitate this level of compromise are the same systems that allow RAID hardware to install software that runs before the normal boot sequence; that software allows RAID configuration before the normal hardware gets configured and gets installed within the BIOS, not the drive-based OS. Such subsystems are also able to wake the system and use its full facilities, including their network cards.

    • Rayne says:

      Warning: L O N G reply

      1) Secretary of State Clinton’s email server — the fuzzing of this server with the multiple hacks of Podesta and the DNC appears deliberate to me as part of a disinformation operation. The point was to make this loose link: Clinton = insecurity and/or co-option.

      The media did an incredibly shitty job ensuring these topics were distinct, but then this, too, was part of the disinformation warfare. This won’t sit well with some readers but it was all too easy for certain white male misogynist members of media to fall into the trap of lumping it all together so that we ended up with “But her emails…”

      Why did “her emails” become such a stink among these same media people when the Bush White House was operating in parallel through the RNC’s system and likely elsewhere? The double standard is right there.

      Disinformation works best when there is a nugget of truth. Nobody looks too closely at why a secretary of state felt they needed their own mail server, though the reasons are right there in the open now. Nobody asks why more than one secretary of state felt they needed non-government email to do their job even though they own it now publicly. You, as a former IT person may feel uncomfortable with what you believe are the security issues involved — but what if some of the worst threats are inside the fenceline?

      2) Russian fuckery disrupting the campaign — think about it: how was the DNC getting reliable data about early voting if both the source of the data and the DNC may have been hacked? Either one of these points would have been problematic alone but both? And they’d had their laptops and desktops swapped out, possibly disrupting perceptions about data integrity on top of it?

      3) Voting machines — I want to know what the Russian(s) did in Georgia when they took their little road trip during November (see US v IRA indictment). Just a coincidence, I’m sure, that both a major voting equipment manufacturer and Equifax are located in Georgia.

      4) Hardware-based vulnerabilities — I wrote a few pieces a while ago about the supply chain hack and other hardware-based vulnerabilities. I received quite a bit of pushback about it. Yeah, well, I notice Bloomberg still hasn’t taken down their reporting on which my pieces were based. This problem hasn’t gotten the attention it deserves; I suspect doing so would create havoc we’re not ready to deal with effectively. The magnitude of the problem is slowly leaking out, though. I just can’t say definitively it played a role in 2016 — there’s not enough data to suggest it when human interference is so freaking obvious.

      • roberts robot double says:

        1) >> but what if some of the worst threats are inside the fenceline?

        That’s an excellent perspective I’ve never considered before, but our ability to trace the actions of our *cough* “public servants” *cough* overrides their need to effect countermeasures against the infighting that goes on within the government. Of course, accountability for the government is our only path to manifesting the utterly fantastic ideals the Founders put on paper; however very much hypocritical they were, they did a good job for all of us, wittingly or unwittingly — it’s just the implementation that’s fscked.

        2) Interesting. I have never contemplated the data flow needs of a high-level political campaign but you paint an even worse picture in this reply than I gathered from your original post. I know from my readings of security for over a decade now that one needs “defense in depth” — the DNC attacks look like their counterpart: “offense in depth”. Kindof impressive, really.

        3) And, IIRC, GA wiped a/some compromised hard drive/s from the 2016 election. Please correct me if I’m wrong.

        4) Yeah, well you have to dig pretty deep into mobo documentation to understand just how devastating compromising System Management Mode is to a system’s security. SMM literally preempts the operating system without an effing trace and has ring negative infinity privileges.

        I have an old OpenBSD box I built circa 2006 whose cdrom drive automatically closes itself as soon as it starts to open, if the button responds at all (it did not do that for its first many years of service). I built it using a bog standard corporate mobo from Intel. Being less infrastructure wise back then I had no idea how many out-of-band bells and whistles Intel builds into those things, but I had not yet contemplated provisioning OS and hardware software updates to thousands of company boxes yet (back then we just each kept our own boxes up-to-date ad hoc as we desired and I was not an system architect yet).

        Thing is, keeping the cdrom from opening is a relatively easy and effective way to keep people from booting from a system tools cdrom that allows things like upgrading the BIOS, dumping the SMM handler code and such (and it was still in the infancy of creating an easily bootable USB thumb drive). Between the fact that it really could be a hardware problem combined with the box continuing to work fine otherwise with a stable installation, I just unplugged it from the internet as it was purely my local software devepment box and I didn’t even run X as my software is text mode command prompt anyway.

        So, it’s either one of those “lie low and gather info” advanced persistant threat type of infestations or a figment of my imagination. I’d love to have the cash to have it analyzed by a really top-notch security outfit but I’m happy to just be on a couple of linux laptops now.

        Anyway, I know that most people called bullshit on Bloomberg’s story, but to think that the Chinese govt doesn’t have assets in place to implement targeted hardware-based malware ops is as silly as believing the NSA is incapable of the same thing. Once they’re in the hardware, the software is putty in their hands and I doubt any consumer OS product has the ability to detect or defend against such subversions. I doubt those organizations will likely do such things to an entire product line or run but that doesn’t mean they don’t have the tech fully tested and ready to go for targeted ops. So, yeah, maybe Bloomberg had some sketchy sources (sprinkling of disinformation perhaps?) but that doesn’t mean that such ops haven’t been carried out.

        And some topics require length, my friend ;-)

    • P J Evans says:

      AFAIK her server was secure and never hacked, so all that noise is just whining for political advantage. (I have a friend who kept his data tape rack in an unused shower next to his home office. It wasn’t easily visible to people, and the easiest way out was through his office.)

      • roberts robot double says:

        a) It may not even be possible to secure a bare, directly-exposed-to-the-Internet Exchange Server installation, given Exchange being a horribly and historically insecure piece of crap combined with Microsoft OSes’ fundamental vulnerabilities and the plethora of side-channel attacks now known to be the direct result of Intel’s processors’ caching architecture.

        b) If such servers are even securable, it takes an elite team of utter badasses to architect the surrounding network security infrastructure, configure the server itself and perform the required maintenance and monitoring to keep the shields up. I didn’t get the impression that HRC or the DNC were so tech savvy to understand how utterly fraught infosec is; if they just hired a guy with his certs to run a server (gotta watch those IT costs! lol), that thing will be very quickly compromised by any number of top-flight hacking groups the world over, from nation-state groups to just good old fashioned criminal orgs. To secure such a system, there is no substitute for my favorite aphorism (from China): “Pay a lot, cry once.”

    • JV says:

      Thanks for the post Rayne.
      During the 2016 email server melee, I tried to explain to people that HRC likely wouldn’t know what a server looked like if it bit her. If there were some at her home, they were likely in a locked cabinet that she didn’t have access to.
      Trump and media minions tried to paint a picture of a technically savvy HRC and staff. In my experience doing tech support/networking, most executives, high level politicians, and their staff, can’t and don’t even setup their own email.
      I don’t have a happy outlook. I see the 2016 election as just another step forward into the “Permanent Republican Majority” envisioned by Karl Rove, et. al and implemented by Citizens United (which Hillary threatened to overturn).
      https://www.democraticunderground.com/100211874474

    • Sofla says:

      There was no bathroom server.

      The company to which Clinton eventually migrated her server after her work need had ended had a former employee who had worked in their original location. Before they moved into larger dedicated HQ operations, they operated out of a small townhouse. At that time she worked there, they had a server or servers in a closet in the hallway, as she recounted in an interview that was reported.

      This former woman employee admitted she had never visited their larger professional HQ where they’d moved long before getting Clinton’s server maintenance contract. She had no knowledge of their server arrangements there. The server in a closet in the hallway was morphed into a server in the bathroom (it was never there), and Clinton’s server got mythically placed into the falsely mangled bathroom server location although their contract with her was some years after they’d moved into standard server farm space.

      Originally, the server was in the basement of her home, but not in the bathroom there, either. It’s a tale that lost all resemblance to reality in the re-telling, like a game of telephone, and once settled into its final form, got burned into popular thinking by repetition.

      • roberts robot double says:

        Thanks for the correction. Yeah, it’s strange, insightful and telling how one simple phrase can get locked into our social consciousness. With so much info hitting us these days, it’s difficult for us to either deep-dive all the stories or retain the details. Yet more praise for Ms. Wheeler et al and this excellent villiage.

        The sad and most salient point in this is that it appears that it really only takes a small number of bad actors to poison the bulk of the entire world internet community, and we are only just barely waking up to this fact in the face of this very new social media technology.

        The fact does remain, however, that securing an Exchange Server in any location — especially when one is or has been the Sec of State — requires a team of very rare and very expensive ninjas. Very few people are technicallly aware enough to understand this fact, especially someone as old as HRC. Running an Exchange Server out of a town house may or may not have been shady, but it was still batshit insane from an opsec standpoint.

        • Rayne says:

          Don’t start with the ageism. Age had little to do with what happened wrt information security. And we as observers with limited access to information can’t be absolutely certain what issues drove the decisions to operate outside .gov servers. After the loss of millions of emails under Bush/Cheney — and email archiving problems going back to 1998 under Bill Clinton — I can’t say I’d have felt comfortable trying to run State Department inside the fenceline.

          • roberts robot double says:

            Is it ageism for me to point out that Trump supporters skew old as fuck?

            Is there an ‘-ism’ to describe how Trump supporters tend to get offended when presented with facts they don’t want to deal with?

            As someone approaching 50, I *KNOW* I’m not the programmer I was at 30. Can I be an ‘ageist’ if I admit it in myself who is not yet old, merely no longer young?

            I apologize, it’s been over 25 years since I could be offended. Most people don’t even begin to realize how their ego manifests itself, especially if they hold a bit of power over the other person.

            The real problem is not old age, but the tendency for people to stop learning, which is especially damning when the person has never cared to learn about themself. “Know thyself”, ya know? Only then can the doorway to Wisdom be stepped through.

            • Rayne says:

              The dig was about HRC’s age. Unless I was supposed to take that as a subtle bash against older women. As I am approaching 60 myself I am not having it.

              • roberts robot double says:

                You know why I can’t be offended? Because I know who I am, foibles and all, and I try to honestly and actively fight them; therefore, if someone says something about the tendencies of human beings, I try to appropriately either dismiss it as irrelevant to myself or honestly own it if it applies.

                Who has time for being offended in this beautiful, troubled world when we are all lucky to be alive, lucky to be able to choose sides to fight against the evil of the world?

                Being a fool has nothing to do with age, but few human beings outgrow their egos and consciously self-evolve into Wisdom.

                You do a yeoman’s job on this important website; you’re obviously no dotard. I absolutely treasure the Wisdom that our open-minded elders can share with us, but they are very, very thin on the ground my friend.

            • P J Evans says:

              and in the context where you put it, it doesn’t come across that way at all.
              Srsly, what does human aging have to do with servers?

              • roberts robot double says:

                >> what does human aging have to do with servers?

                The vast majority of people, as they age, become less likely to admit that they don’t know certain things and are thus less likely to listen to others’ advice. Perhaps like, “Maybe you should go to the upper midwest and do some campaigning.”

                Also, treating IT like it’s the 90’s means hiring a single person to set up a personal Exchange Server because of ill-founded reasons, instead of grokking that 2015 is a dangerous-ass place to deploy a friggin Exchange Server in a town house.

                Look, my parents won’t listen to shit about Trump’s various flavors of evil, or the Catholic Church’s, for that matter. That is because most people become less pliable as they age and thus become entrenched in their own illogically-based self-confidence, which usually means overinflated ego. That attitude leads to problems in the young and old. But if you’re the avowed enemy of the most ruthless dictator on Earth who happened to have amassed arguably the greatest infrastructure of hackers the world has ever seen, well, such ignorance of opsec can be fatal to one’s campaign.

                It’s really important to know what one doesn’t know and most people begin failing at that at a very young age, and that failure becomes only more pronounced as they age. Like all personality defects that we don’t actively fight, they win more and more.

  10. Ed Walker says:

    Very helpful explanation, Rayne, and good point about the ignorance of the base. It might be helpful for those active on social media to use this to reply to people who seem not to have this basic information. You can find them on twitter in replies to tweets from Elizabeth Warren, AOC, and other progressives. Remember to be polite about it. It’s not their fault they don’t know this stuff, and it’s really even more complicated than Rayne explains.

    My son’s firm uses AWS for complex computing. The system allocates a fixed block of cores to the firm, and if they exceed that usage, it provides more on the fly. Think of cores as that figure on the chip on your device. I use an old MacPro with two, but current Apple chips for phones have six. Figuring out which cores his code uses is possible, I suppose, but pointless. At least that’s how I understand it.

    • Rayne says:

      A fixed block can still be virtualized — a user assigned a fixed block ideally never knows the condition of the hardware supporting that block. The beginning and ending addresses of the block aren’t shared and can change depending on where the block is parked at any given time.

  11. Hops says:

    I don’t think they get a “fixed block” of cores per se, but a fixed number. They wouldn’t let cores sit idle and most customers aren’t using them 24/7.

    For more information about cloud computing, check out Hadoop on Wikipedia. Even the files get distributed.

    • Ed Walker says:

      Yes, that’s correct. I should have said they guarantee a fixed number of cores will always be available; that’s one price, and overages are another price. Part of the work involves making sure normal work of the business stays inside the lower guaranteed price structure.

  12. PeteT says:

    Well done Rayne.

    I am a former computer tech who worked with highly available systems (clustering) , disaster recovery, etc but just before or as the cloud based infrastructure was taking hold. First there was host-based clustering, then VMware-like clustering and now…in the cloud.

    Even today, with security uppermost in my mind, for my personal and small business needs I struggle with using localized programs, etc versus something along the lines of a chromebook. Heavens no, not going to align myself completely with the Google monster…he says as yesterday I transferred the gasps of a dying (now dead) eight year old MacBook Pro to a new iMac.

    Just dunno if I want to run certain personal/business accounting software in the cloud (e.g. QuickBooks) or not. Word processing, spreadsheets, and does anyone do powerpoint anymore – not this guy cause you don’t need Microsoft for that anymore even on the desktop.

    Smart desktop or chrome-like-book. And, even though I eat a bit of the Apple, the iPad is not very good at all for the not-the-desktop replacement.

    Pete

    • JamesJoyce says:

      On a prior site I once commented. Those comments cost me three laptop computers. They were targeted.

      Ball bearing plants were at one time important military targets.

      Nothing rolls well when ball bearings are scarce.

      Likewise it is hard to communicate or run any campaign with infected computers just like a shortage of ball bearings.

      Yesterday I witnessed a person attempt to check engine’s oil level when his car’s motor was running. Never mind understanding the concept of “cloud” or a network of interconnected “servers,” or female physiology?

      Darwinian Chum came to mind as I looked at this sad chump, thinking of the phrase: “…below-par, for the course.”

  13. oldpaint says:

    The server? What about the stealth airplane? It’s invisible, right? You can’t see it! And the Mueller report — it cleared Trump!

    All of this lunacy is deliberate, and it didn’t start with Trump. Sometimes I think the “conservative movement” adopted Orwell’s 1984 as a textbook, along with Ayn Rand’s novels. Just waiting for Trump to announce the creation of the Ministry of Truth.

      • JamesJoyce says:

        Bill Lee would better.

        He has experience being the real Spaceman.

        Maybe he would seek Mickey River’s advice on future policy?

      • Troutwaxer says:

        Naw, it will be William Shatner. You don’t think he’s got a purpose for all those wingnut tweets?

  14. OldTulsaDude says:

    I’m sure Mick will hate this:

    I see our border and I want a wall that’s black
    No colors can come in, no browns, no tans, no blacks
    Build the wall and put black spikes on top of it, too
    If you don’t believe I can do it
    Better take it up with John Yoo

  15. ecsCoffee says:

    Hi Rayne, this was super-helpful for someone like me who is far outside the information technology field. Thank you for your clear introduction to the material.

Comments are closed.