Information in Amended DNC Lawsuit Reveals that Roger Stone Is at Significantly Greater Risk for CFAA Indictment

Back in November, I wrote a post considering whether Roger Stone could be charged in a CFAA conspiracy. I noted that the last hack noted in the GRU indictment may have post-dated communications Stone had with Guccifer 2.0, in which Stone scoffed at the analytical information released as part of the DCCC hack. I pointed to this passage from the GRU indictment, showing that the GRU hack of the DNC analytics hosted on an AWS server may have post-dated those conversations between Guccifer 2.0 and Stone.

I’m writing a response to the Wikileaks defense against the DNC lawsuit for its involvements in the 2016 election attack, and so have only now gotten around to reading the amended complaint against Stone and others that the DNC filed in the wake of the GRU indictment. And it reveals that the AWS hack was far worse than described in the GRU indictment — and it continued well after that Stone conversation with Guccifer 2.0.

None of this long passage is footnoted in the complaint. It has to be based on the DNC’s own knowledge of the AWS hack.

On September 20, 2016, CrowdStrike’s monitoring service discovered that unauthorized users—later discovered to be GRU officers—had accessed the DNC’s cloud-computing service. The cloud-computing service housed test applications related to the DNC’s analytics. The DNC’s analytics are its most important, valuable, and highly confidential tools. While the DNC did not detect unauthorized access to its voter file, access to these test applications could have provided the GRU with the ability to see how the DNC was evaluating and processing data critical to its principal goal of winning elections. Forensic analysis showed that the unauthorized users had stolen the contents of these virtual servers by making exact duplicates (“snapshots”) of them and moving those snapshots to other accounts they owned on the same service. The GRU stole multiple snapshots of these virtual servers between September 5, 2016 and September 22, 2016. The U.S. government later concluded that this cyberattack had been executed by the GRU as part of its broader campaign to damage to the Democratic party.

In 2016, the DNC used Amazon Web Services (“AWS”), an Amazon-owned company that provides cloud computing space for businesses, as its “data warehouse” for storing and analyzing almost all of its data.

To store and analyze the data, the DNC used a software program called Vertica, which was run on the AWS servers. Vertica is a Hewlett Packard program, which the DNC licensed. The data stored on Vertica included voter contact information, such as the names, addresses, phone numbers, and email addresses of voters, and notes from the DNC’s prior contacts with these voters. The DNC also stored “digital information” on AWS servers. “Digital information” included data about the DNC’s online engagement, such as DNC email lists, the number of times internet users click on DNC advertisements (or “click rates”), and the number of times internet users click on links embedded in DNC emails (or “engagement rates”). The DNC also used AWS to store volunteer information—such as the list of people who have signed up for DNC-sponsored events and the number of people who attended those events.

Vertica was used to both store DNC data and organize the data so that DNC computer engineers could access it. To use the Vertica data, DNC employees could not simply type a plain-English question into the database. Instead, DNC engineers needed to write lines of computer code that instructed Vertica to search for and display a data set. The computer engineers’ coded requests for data are called “queries.”

When the DNC wanted to access and use the data it collected, the DNC described the information it wanted to retrieve, and DNC computer engineers designed and coded the appropriate “queries” to produce that data. These queries are secret, sensitive work product developed by the DNC for the purpose of retrieving specific cross-sections of information in order to develop political, financial, and voter engagement strategies and services. Many of these queries are used or intended for use in interstate commerce. The DNC derives value from these queries by virtue of their secrecy: if made public, these queries would reveal critical insights into the DNC’s political, financial, and voter engagement strategies. DNC computer engineers could save Vertica queries that they run repeatedly. In 2016, some of the DNC’s most frequently used Vertica queries—which revealed fundamental elements of the DNC’s political and financial strategies— were stored on the AWS servers.

When the DNC wanted to analyze its data to look for helpful patterns or trends, the DNC used another piece of software called Tableau. Tableau is commercial software not developed by DNC engineers. Instead, the DNC purchased a license for the Tableau software, and ran the software against Vertica.

Using Tableau, the DNC was able to develop graphs, maps, and other visual reports based on the data stored on Vertica. When the DNC wanted to visualize the data it collected, the DNC described the information it wanted to examine, and DNC computer engineers designed and coded the appropriate “Tableau queries” to produce that data in the form requested. These Tableau queries are secret, sensitive work product developed by the DNC for the purpose of transforming its raw data into useful visualizations. The DNC derives value from these queries by virtue of their secrecy: if made public, these queries would reveal critical insights into the DNC’s political, financial, and voter engagement strategies and services. Many of these queries are used or intended for use in interstate commerce.

DNC computer engineers could also save Tableau queries that they ran repeatedly. In 2016, some of the DNC’s most frequently used Tableau queries—which revealed fundamental elements of the DNC’s political and financial strategies—were stored on the AWS servers.

The DNC’s Vertica queries and Tableau Queries that allow DNC staff to analyze their data and measure their progress toward their strategic goals—collectively, the DNC’s “analytics,”—are its most important, valuable, and highly confidential tools. Because these tools were so essential, the DNC would often test them before they were used broadly.

The tests were conducted using “testing clusters”—designated portions of the AWS servers where the DNC tests new pieces of software, including new Tableau and Vertica Queries. To test a new query, a DNC engineer could use the query on a “synthetic” data set—mock-up data generated for the purpose of testing new software—or a small set of real data. For example, the DNC might test a Tableau query by applying the software to a set of information from a specific state or in a specific age range. Thus, the testing clusters housed sensitive, proprietary pieces of software under development. As described above, the DNC derives significant value from its proprietary software by virtue of its secrecy: if made public, it would reveal critical insights into the DNC’s political, financial, and voter engagement strategies and services, many of which are used or intended for use in interstate commerce.

The DNC protected all of the data and code in its AWS servers by, among other things, restricting access to authorized users. To gain access to the AWS servers themselves, an authorized user had to take multiple steps. First, the authorized user would have to log onto a Virtual Private Network (VPN) using a unique username and password. Second, once the user entered a valid and password, the system would send a unique six-digit code (PIN) to the authorized user’s phone, and the user would have 30 seconds to type it into the computer system. This two-step process is commonly known as “two-factor authentication.”

Authorized users would also employ a two-factor authentication system to access Tableau visualizations. First, they would log into a Google account with a unique username and password, and then they would enter a pin sent to their cell phones.

Finally, the DNC’s AWS servers were protected with firewalls and cybersecurity best practices, including: (a) limiting the IP addresses and ports with which users could access servers; (b) auditing user account activities; and (c) monitoring authentication and access attempts.

On September 20, 2016, CrowdStrike’s monitoring service discovered that unauthorized users had breached DNC AWS servers that contained testing clusters. Further forensic analysis showed that the unauthorized users had stolen the contents of these DNC AWS servers by taking snapshots of the virtual servers, and had moved those replicas to other AWS accounts they controlled. The GRU stole multiple snapshots of these servers between September 5, 2016 and September 22, 2016. The U.S. later concluded that this cyberattack had been executed by the GRU as part of its broader campaign to damage to the Democratic party. The GRU could have derived significant economic value from the theft of the DNC’s data by, among other possibilities, selling the data to the highest bidder.

The software would also be usable as executable code by DNC opponents, who could attempt to re-create DNC data visualizations or derive DNC strategy decisions by analyzing the tools the DNC uses to analyze its data. [my emphasis]

In other words, at least one of those snapshots was stolen after Stone suggested he would like better analytics data than what GRU had publicly released via HelloFL. So he can no longer say that his communications with Guccifer 2.0 preceded all the hacking. Which the nifty timeline Stone’s attorney submitted in conjunction with his motion to dismiss doesn’t account for at all.

Given Stone’s history of non-denial denials for crimes he commits, I’d say this stunted timeline doesn’t help him much.

Here’s Stone’s motion to dismiss. As with his nifty timeline, he does not address — at all — the communications between him and Guccifer 2.0 regarding analytics. It does, however, include this tagline.

He is the First Amendment running, not walking; but his conduct cannot be adjudged a civil wrong.

Past history says Stone’s rat-fuckery tends to be easily found in his swiss cheese denials, and I’d say this is one example.

Note that, a week after DNC submitted its amended complaint on October 4, WikiLeaks released a proprietary AWS document showing the locations of all AWS’s servers around the world — something that is not all that newsworthy, but something that would be incredibly valuable for those trying to compromise AWS. That was one of its only releases since the crackdown on Assange has intensified.

As I disclosed in July, I provided information to the FBI on issues related to the Mueller investigation, so I’m going to include disclosure statements on Mueller investigation posts from here on out. I will include the disclosure whether or not the stuff I shared with the FBI pertains to the subject of the post. 

image_print
185 replies
  1. Greg says:

    Not that I have any particular motivation in coming to the defense of AWS, but as someone who uses AWS and is loosely familiar with many of the technologies described in the complaint, I find the repeated references to AWS to be a bit off target (at least potentially).

    Depending on the nature of how the GRU gained access to the DNC servers and copied/snapshotted them, I wouldn’t refer to this as an “AWS hack” or anything along those lines. When setting up servers on AWS or similar services, you are fully responsible for virtually all aspects of the security of the systems. Amazon/AWS gives you a variety of tools, but it’s up to the customer to properly implement them and incorporate them into their broader security strategy.

    Unless the GRU actually gained access to proprietary AWS systems, or manipulated AWS personnel in some way to aid their attack on the DNC, I wouldn’t consider the fact that AWS happened to be the service provider to be much more than a footnote. The best analogy I can think of is if I left my laptop open on my desk and someone broke into my home and got information off of it. In this scenario it would be bizarre/uninteresting to repeatedly refer to it as a DELL laptop or an HP laptop in my criminal proceedings. What matters is I had a laptop, it got improperly/criminally accessed, and it’s besides the point what make/model it was.

    An exception/counter to what I’m saying is the proprietary AWS document that Wikileaks released. I don’t know if that’s in any way connected to *how* the GRU accessed the DNC servers on AWS. Maybe it is, and maybe furthermore the GRU truly did infiltrate AWS proprietary systems, etc. But if that were the case I think it would be a much bigger story in the tech world.

    Anyway, my primary reason for this long note is that for all those who are not very technical or familiar with cloud services like AWS, that the repeated references to AWS in the complaint would make it seem like they are the ones responsible for this security failure or that their products are deficient in some way. And it’s not clear to me that’s the case at all.

    • bmaz says:

      Actually, “Greg”, I am going to bet you do, indeed, have some focused motivation to so strongly come out of nowhere with your diatribe. Would you like to elaborate on that, or continue to be duplicitous to this forum?

      • Greg says:

        Uhh, no I really don’t have any focused motivation. I’ve been a reader here for over a year but have had nothing to offer in terms of the great legal and strategic analysis that Marcy and others are able to provide. I voraciously consume everything written on here and love the site.

        I just think, exactly as I tried to lay out in my post, that a lay-person might read the COMPLAINT (not Marcy’s post necessarily) and conclude that we must legislate AWS and similar service off the face of the planet. And I think that would be a bad conclusion, so I tried to give my opinion.

        What makes you think I’m being duplicitous? What makes you think my name is not Greg (it is)?  I follow you on Twitter because I enjoy reading your takes on things, but I also see that you’re often really mad at all sorts of people. Jeez.

        • Michael Keenan says:

          I noticed too. Is that maybe why because I mentioned Ray McGovern that I can no longer follow emptywheel on twitter? @CitizenMichael1 Not really sure why but can I face my accuser?

      • phazed says:

        I’m a lurker without an agenda. I’m not going to bat for AWS or whatever the opposite of AWS is. I use a mixture of VPS providers, and work for none of them.

         

        To me, this doesn’t “read” like AWS is being blamed at all. In fact the way I read it, Crowdstrike makes no such claim, and probably assumes nobody else will either.

         

        There is a bit of a disconnect here though, as the things stolen and the security implemented to protect them, is not the security which was breached.

         

        “the unauthorized users had stolen the contents of these DNC AWS servers by taking snapshots of the virtual servers, and had moved those replicas to other AWS accounts they controlled”

         

        This means the hackers gained access to DNC AWS resources using DNC administrator credentials via either the AWS console or APIs. There is a non-zero probability that they did this by compromising AWS systems themselves, however unlikely. Access to a DevOps or Software engineer’s machine is all that is required to achieve this, and given that we know Cozy Bear at the very least had such access, I suspect this is the conclusion Crowdstrike has drawn as well.

    • earlofhuntingdon says:

      This is not your average web site regarding either its content or its commentary.  Your claim not to have a “particular motivation” to comment on behalf of AWS is a non-denial of a commercial connection with it.  If you have one, you should disclose it.  Failure to do so will not improve AWS’s reputation or yours.

      • Greg says:

        I have no commercial connection to AWS.  I am a tiny, tiny customer of theirs, and many other cloud services. I spend about $125 per month on AWS. I have no loyalty to them whatsoever. In fact, I take issue with many, many things that their parent company Amazon does.

        All I’m saying is this complaint (not written by Marcy!) reads as showing some negligence and fault from AWS. And for anyone who is not a “computer person” they might draw conclusions that I think could be incorrect. So I thought I would try and share my point of view.

        • bmaz says:

          This forum is most certainly not Twitter. We are just an honest little blog.

          And I’d love to hear your explanations, but am not buying your purported “motivation”.

        • Greg says:

          bmaz, I know all about this blog. It’s one of my favorite sites for the past year of my life. I read *every* article here. And finally on this article I thought I could chime in and help give some perspective, because this is something I know a little bit about (and it’s really just a little. Marcy regularly engages actual expert infosec people on Twitter, and I’m far, far from that). But I think I can still add value.

          If you actually read my post again, I think it’s pretty even-keeled. It might seem strange that I jumped on this AWS thing, but AWS was mentioned 16 times in the post. I thought it was appropriate to comment on.

        • bmaz says:

          You have been here for eight impertinent and belligerent comments over four hours and think anybody here is going to email you with a discussion? Are you high?

        • Greg says:

          Dude, I don’t know what else to say. I don’t work for Amazon or have any kind of ulterior motive. I don’t want to hijack the comments of this article so I’ll just move on after this. Feel free to email me and I’ll try to convince you I’m just a regular reader. Otherwise have a good one.

        • Justin says:

          I’m also a long time lurker on this website which I started reading diligently after hearing Marcy’s first interview on Pod Save America. I don’t think Greg was out of line, he was just posting his thoughts in the comments. bmaz’s reaction really doesn’t seem warranted and frankly it seems paranoid. I mean unless his email is [email protected] because then I get it.

          I also get that the bloggers at emptywheel should be paranoid based on their efforts at uncovering the truth. But it seems a bit of an over reaction.

          @Greg, I think that AWS is mentioned so often because these things need to be explained to the courts directly. I didn’t read Marcy’s article thinking that these things were in some way AWS’s fault.

        • bmaz says:

          Golly Justin, thanks for clocking in from the other side to the world (Germany is it?). You have anything else to offer in support of the troll “Greg” that has suddenly lobbed in despite his sudden masked appearance in the last couple of hours?

          As with “Greg” I will await your substantive contribution as opposed to trolling in like authoritative assholes.

          Let nobody who is a regular here mistake these charlatans for anything more than interloper trolls. That is exactly what they are.

        • orionATL says:

          and neither do i, justin. i appreciate greg’s input. it is substantive, which is more than i can say about the carping criticism directed at it.

        • taluslope says:

          I am a software professional and see nothing nefarious in Greg’s post.  I enjoy seeing experts in legal matters post here but also appreciate thoughtful comments from those in other fields.

        • readerOfTeaLeaves says:

          bmaz, I’m mystified.

          Perhaps you know something from ‘Greg’s ISP or other background info that  I don’t.

          However, I read this post the same way that Greg did – as something that a person who doesn’t use AWS technologies, nor Tableau, might think is an ‘Amazon’ problem.  Or a Tableau problem.  It isn’t.

          Also, every time someone types a Google search, they are performing a ‘query’ for Christ’s sake.  It’s not rocket science, even if it’s a SQL query.  The amount of mystification and mumbo-jumbo and claims of valuable intellectual property related to queries can really be irksome to read about some days.   It’s made fortunes for some IP lawyers, but it has also inflated way too many egos and fed way too much vanity IMVHO.  As that has happened, the whole ‘valuable queries/IP’ topic has become a gold mine for enough folks that it has shaped a whole culture, which has some downsides.

          IMVHO, Greg is trying to help demystify the bullshit factor around ‘queries’ and AWS, and I happen to think that is a good thing.  If he’s been reading here awhile, he can tell that there are some *very* smart, very technical commenters (Frank Probst, et al), who are not at all bamboozled by technical content.  I would argue that EW’s ability to attract (and retain) smart, technically proficient readers and commenters has really set this blog apart (in a good way) through the years.

          However, there are some newer readers still on the learning curve, and they should not be bamboozled, nor intimidated, by technical bullshit.  In the sense of trying to get *everyone* up to speed on the same page, and make clear what a venomous traitor Roger Stone actually is by being using hacked info, IMVHO Greg is doing the blog readership a service.

          Unfortunately, some GRU hackers got inside AWS servers. That’s a problem.  Any role that Roger Stone had in enabling that behavior, paying for it, cooperating with it, or using the information is beyond reprehensible.  It doesn’t matter whose server, what query language, or what technologies were involved — Roger Stone is a viper of the lowest phylum.

          But with respect to AWS servers, millions of us use them all day long and are never even aware of it.  Netflix uses AWS servers; so does Kellogg’s breakfast cereal company.  Just google ‘businesses using AWS‘ and  you’ll get more info than you can read.  This is a big, quiet piece of Amazons’ business, and I would not want to be the asshat who tried to break into their servers, partly because it’s stupid, but also because it’s sinister.  From a practical perspective, you are screwing with a lot of rice bowls, a lot of businesses, and a lot of governments.  I don’t even have words for how stupid that would make you.

          Honestly, if Greg lived in my neighborhood, his comment would be read like, ‘uh huh’.  One neighbor works for Tableau, another for Amazon… well, you get the idea.  It’s a very diverse, very ‘global’ workforce.   We could come up with a new definition of ‘stupid’ as ‘hacking AWS’ and I’d be totally fine with that criteria.   I hope it becomes a global meme.

          I read Greg’s comment as an effort to try and demystify the terms ‘query’ and AWS, so that people see much more clearly that the technologies were named in the legal documents as a way to be very specific, grounded, and legitimately document what has happened.  However, for blog readers, don’t get too hung up on the tech stuff – the main point is that we have had, associated with the GOP for decades, Roger Stone, his pal Manafort, and their ilk who are so treasonous, so treacherous, so venal that they are willing to use *whatever* technologies they can to cooperate with foreign interests in order to subvert American laws, governance, and civil society.

          When I think how many people are floating around on aircraft carriers, or volunteering at food banks, or tutoring kids, it makes my blood boil to think that all these decent people are being subverted by the likes of Roger Stone.

          Honestly, I think that Greg has his eye on the ball in the sense of trying to bring other readers along and make the main themes clear.

           

          ——————————

          *tiny disclosure: I was an early AMZN employee.  Back in the day.  I  don’t work there now, don’t own stock, and don’t have any financial or employment dog in this argument.  I will admit that I know several AMZN employees, and they are among the smartest, hardest working, most decent people of my acquaintance.

        • hester says:

          A humble thank you, readerOfTeaLeaves  for your comment.  Well said and much appreciated.  I read Greg’s comment as informative.  It was somehow  interpreted as below board and other than what it appeared to me to be.

          I tend to be intimidated by some of the staff / posters here so i generally refrain from commenting.   Nothing I have to say is ever earth-shattering so no loss for anyone.

          Thank you again

        • readerOfTeaLeaves says:

          Well, you have a front row seat if Bmaz chews my ass about this.  But honestly, I don’t know how he, Rayne, Ed, and Jim keep the trolls down, so I cut him a whole lot of slack.  The more threatening the Mueller investigation becomes, the more trolls will turn up and the harder it will be to keep them at bay: I’d prefer that he overreact, as he and I have both seen blogs that went south because they were overtaken by trolls.

          I appreciate your comment, and would encourage  you not to be intimidated.  I’ve always felt that EW’s blog was a place for learning, where people can become empowered by having better information, more clarity, and a sharper focus.  To me, your comment abundantly expresses that spirit.

          Ask questions when you need to, or do a google search to figure out terms or topics that you don’t yet understand. The longer I live, the more I value inquiring minds like those that read and comment around here.  (Even when Bmaz chews my ass, but let’s keep that on the QT ;-)

        • Cicero101 says:

          I’m massively impressed how free of trolls etc this site is, and deeply grateful. Thanks to those doing that.

    • Michelle says:

      Or the repeated references to AWS are because the DNC suffered hacks to multiple server networks and calling the hack of their cloud hosting the “DNC AWS hack” is a straightforward way of distinguishing it from the hack of, say, the DNC’s local Microsoft Exchange server.

      • Greg says:

        That’s true, but part of my point is that in this day and age, it doesn’t really matter if it was a local Exchange server in the coat closet of their offices, or a virtual private server in AWS’s warehouse. The DNC has an essentially equivalent role in managing both of those two things.

        I think the average reader would look at all of this and conclude AWS really screwed up here. And I am just trying to preemptively give some counter argument.

        • P J Evans says:

          Explain how you think that AWS has any reason to protect its customers from hacking by foreign powers.

          (Also, I’m reading between the lines of bmaz’s comments that you made the mistake of changing your screen name here after your first or second post didn’t get the reaction you wanted. Were you the one using “Tech Support”?)

        • Greg says:

          I really don’t think I changed my screen name or did anything shady, at least not purposely. Every time I post a comment, I have to type in a name by hand, so I type Greg. It’s my real name, I promise.  And every time I have to put an email address. It’s my real email address, I promise.  All of this has been done through the same ol’ Comcast internet connection from my home today.

          I get that there are a lot of people trying to screw with this website and that bmaz and any other moderators need to play tough. But it’s certainly not what I’m up to.

      • Tech Support says:

        @Michelle this actually gets at the heart of what the complaint doesn’t answer

        First off this isn’t strictly correct. You are conflating the technologies being deployed (Tableau/Vertica vs. Exchange) and the location those services are being hosted (cloud vs. “on prem”). Although Exchange in the cloud usually means Microsoft’s Azure via Office365, it’s entirely possible that you could deploy “on prem” Exchange to a virtual server hosted in AWS.

        The biggest risk for misunderstanding in calling this attack an “AWS” hack is what it implies about the depth and breadth of what the attackers actually compromised. The implications are huge.

        For example let’s suppose that the DNC servers were compromised because the credentials of an engineer with access to those servers were obtained via spear fishing, malware, harvesting re-used credentials from third-party hacking target… whatever.  The attackers would have gained access to those specific systems and not necessarily anything else hosted on AWS.

        On the other hand, if an “AWS hack” refers to the AWS infrastructure itself being compromised, then that implies the possibility that the attackers can gain access to multiple customers of AWS, maybe even all of them depending on what exactly has been exploited. That’s an incredibly important distinction and not one that is not at all clear.

        FWIW, I’d imagine that it was the more limited scenario in this case… mainly because if the production servers with the real voter data were also hosted on AWS, and they could reach it via AWS infrastructure, then you’d think they’d skip the test cluster entirely. If that’s true then yeah, calling it an “AWS hack” is an overstatement.

        • timbo says:

          I am reminded of the compromised Intel chipset and processor problems that became public at the end of 2017…  For how long were those exploitations available to intelligence services/black hats prior to that?

          So, why not just use the Test Cluster anyways?  Indirection involves not revealing full capabilities in an intelligence operation, same goes for sophisticated hacking of any live system, as well as live security measures.  Certainly a complete compromise of AWS would fall under the category of hacks that would be a closely guarded secret by hackers if not uncovered by AWS security operations quickly…

        • Tech Support says:

          Two ideas you explore here that I want to break out. First regarding the chipset issue, you ask an unanswerable question. However I thought it was worth pointing out that this hardware exploit was conducted by the Chinese who used their supply chain power to slip the compromised hardware into their targets environments. This is most likely separate from and not beneficial to the GRU agents who attacked the DNC.

          Regarding the possibility that the test cluster was deliberately chosen to avoid revealing the (hypothetical) compromise of the entire AWS environment? Why not hit the production environment and JUST the production environment? That would be no more indicative of an AWS-wide exploit than just hitting the test cluster.

          P.S. Hat tip to @Alan for calling out the opportunity to defeat the cel phone based 2Fa via the use of a Stingray device. He mentioned this in the comments of a more recent post, and I’d agree (with my own limited expertise) that this seems like the most economical, low-effort way to get direct entry into that test cluster by stealing the credentials of an engineer with access to that particular environment.

        • Greg says:

          Enjoyed reading all your comments, @Techsupport! Honestly the technical details in the complaint aren’t making much sense to me, if we really wanted to try and dig into them. I added some more thoughts here, for those that are motivated enough to keep talking about this side of the conversation:

          https://www.emptywheel.net/2018/12/09/information-in-amended-dnc-lawsuit-reveals-that-roger-stone-is-at-significantly-greater-risk-for-cfaa-indictment/#comment-762654

        • Michael Keenan says:

          So at risk of getting an ass chew by the Bmaz what of Binneys claim of a leak over a hack given the technical feasibility of one by memory stick over a straight internet raid? It is kind of hard to ignore this guy given his standing. What saith the Tech Support guy?

    • greengiant says:

      There is already no end of commentary available ( elsewhere) describing security problems with cloud based computing and data bases. The market leader AWS and others are a primary target. Companies and the DNC that are cost driven can unkowningly or just don’t give a damn to make the tradeoff. The least they could do is encrypt all data no matter where stored to add one more penetrable ring of defense. No upside to victim blaming in this political and foreign invasion of the US.

      • orionATL says:

        greengiant –

        thanks. i suspected what you have said:

        “there is already no end of commentary available ( elsewhere) describing security problems with cloud based computing and data bases. The market leader AWS and others are a primary target.”

        but don’t deal with this stuff professionally.

    • eh says:

      I get the point you’re trying to make, but it’s my understanding that at the end of the day it doesn’t matter whether it was Amazon internal systems that were hacked or it was a DNC instance running on AWS as far as CFAA charges go.

    • pseudonymous in nc says:

      I’m not going to join the pile-on.

      There are two obvious possibilities here. The first is that the AWS control panel didn’t have the same auth controls, IP restrictions and 2FA as the actual Tableau/Vertica server instances. In that case, the GRU could potentially gain access to the control panel (and then snapshot and exfiltrate the EC2 instances) through spearphishing or social engineering. That’s one problem of cloud hosting: the security model for the hosted servers isn’t necessarily the same for the hosting provider’s control panel.

      If the Dems’ AWS control panel did require a specific VPN IP address and 2FA etc. then you have to wonder if there were vulnerabilities in the control panel architecture that could be exploited to grant privileges on other users’ services.

      Maybe it is, and maybe furthermore the GRU truly did infiltrate AWS proprietary systems, etc. But if that were the case I think it would be a much bigger story in the tech world.

      This was also during a period of multiple Xen privilege escalation CVEs, where AWS customers (and customers of other cloud providers) had their virtual servers rebooted so that the host machines could be patched. In late 2017, Amazon announced that it was moving from Xen to KVM as its hypervisor, which raised some eyebrows, as it had developed EC2 on Xen and customised it to the requirements of AWS for over a decade. Exploiting that in a way that targetted a specific guest would be very tricky, but it’s not out of the question.

      • phazed says:

        They could have performed the snapshots remotely from an engineer’s machine. The machine may have had cached 2FA tokens etc..

        Note that they specify that the snapshots were made by the unauthorised users, while also stating that they had user monitoring in place. This would have alerted the user to a spurious 2FA signon, which I’m assuming in this case didn’t happen.

         

        • timbo says:

          Or maybe it did happen but was ignored… because of poor account discipline. Many IT people get sloppy quickly when they’re constantly having to drop whatever they’re doing to login to accounts just to give access to someone else to a command structure, logs, etc.  It’s not uncommon for someone who is not the assigned account owner to have access to the account login procedures.  It’s not a good practice but it something that happens all the time.  So, if the account owner gets a notification that the account has been accessed, it might not be immediately apparent whether the access was “unauthorized” vs some IT dude just not wasting the account owners time by bothering them again to login to do ‘authorized’/legit work.

    • J. H. Frank says:

      Not that I have any particular motivation in coming to the defense of AWS, but as someone who uses AWS and is loosely familiar with many of the technologies described in the complaint, I find the repeated references to AWS to be a bit off target (at least potentially).

      The sticking point here, and all the subsequent arguing, come down to “AWS hack” meaning one thing in the CompSci/OpSec world and something entirely more general when in a legal filing before a judge that may not even own a smartphone.

      It’s like someone with a Doctorate in Latin coming into the comments to explain what Latinate legal terms well, actually mean.  That’s cool, but it’s not exactly relevant to the subject at hand.

      And now bmaz is going think I’m a troll because this is my first ever comment here and I’ve got a gmail address.  (I do comment under the same name over on LGM and on *shudder* reddit, though.)

  2. orionATL says:

    after reading the very detailed passage about the dnc analytics and the security steps the dnc computers techs took, i wonder how anyone could reasonably complain, as happened often in 2017, that the dnc was sloppy in protecting its data and analytical software and oblivious to the possibility of an attack? in fact, my question after reading this is: could anyone other than dnc software engineers have accessed the data contained within the stolen servers?

    did the russians who stole the virtual servers, or the trump campaign actually use it prior to the election?

    • arbusto says:

      It appears DNC IT shop used prudent safeguards but forgot that dealing with the RNC, let alone possible foreign actors, prudent doesn’t hack it.  Also, as shown by the phishing results, the users are the weak link in any security policy.  Hell, short of running analytics on an an isolated computer, in a basement with armed security, in a Faraday cage with independent power the data is still hackable, not to mention offshore manufacturers slipping data gathering ic’s on mother/daughter boards or other hardware.

      • J R in WV says:

        This is all true. There is a reason all national security servers are air-gapped, as in not connected to any networks at all, ever. The air gap refers to the lack of a physical connection to a network.

        I was a software developer as my career, for many years. Developing queries was an important part of my work life. We never used two-part authentication, but we did have several layers of security.

        I don’t think “Greg” is a troll judging from his comment content — looking at his IP address etc would provide way more information, but just what he said isn’t negative or trollish. I don’t much like the philosophy of Amazon either, but I don’t think their commercial operating philosophy is related to the security of their commercially provided systems.

         

        Just my two bits on the security involved. No system connected to any network is really secure, for mission critical systems the actual databases and software repositories need to be completely separate from networks, Air-gapped in other words, as NSA and other intelligence related systems are.

        • Thomas Paine says:

          Amen.  Nearly all govt. systems containing national security information operate as “air gapped” systems for the reasons you cite.  When I was in this world, the “air gap” had to be several feet or more so that NOT ONE STRAY BIT COULD HOP ACROSS THE GAP.   Intranet-connected hardware could not be placed close to even an analog telephone.

          The Web really is a cesspool and there are uncounted bad actors up on it ALL the time.  AWS is out of their element in trying to defend against all this bad stuff.  If you REALLY want security you have to avoid the Web like the plague and set up your own hard-wired Intranet with alarmed conduit and /or NSA-provided encryption hardware and software.

          Since both the DNC and the RCCC have been successfully hacked by foreign actors from outside the USA, the govt. should REQUIRE that sensitive political party information that is national in nature, (including, particularly data analytics), that would be valuable to an foreign agent in perpetuating fraud during an election, should be treated as protected national security information and only hosted on NSA approved systems as a matter of law.  Anything less invites the 2016 and 2018 election hacks all over again.

        • arbusto says:

          My mind boggles that key government agencies especially the CIA and DoD outsource any IT function, let alone use the cloud.  God only knows how complicated a RFI or RFP would be and I sure as shit would enforce the Buy America clause and demand on shore manufacture.  Instead some dipshit Administrator(s) opened up a world of hurt for our security.

  3. DannyD says:

    Super interesting, the ability to exfiltrate AWS EC2 snapshots would seem to indicate that they had access to AWS admin console.  From there, they could have added any IP addresses they wanted to the security groups, replicated anything they wanted, even replace the real DNC data with false data to ‘blind’ the data scientists.

    • Tech Support says:

      even replace the real DNC data with false data to ‘blind’ the data scientists.

      Except as far we seem to know they only got the test environment. So if they had access to the admin console it sounds like they were still restricted in terms of what servers they had the privileges to manipulate.

      • DannyD says:

        Ok, I’ll stipulate to that, as I don’t know either way…but even on a testing cluster, you could induce a situation where the data scientists are blind to a problem, because if you change the data-set that the scientists are using for model validations, they might think things are OK when they’re really not.

        Overall, I just think it’s pretty interesting and ingenious approach to simply ‘snapshot’ an instance instead of trying to break in through the on-machine security.  Sort of like bank robbers taking both the ‘safe’ AND the ‘money’ instead of trying to crack the safe on the premises.

        It’s also interesting, because regardless of the ‘user’ level protections they added to the instance (VPNs, access ‘to’ the instance via limited IP Address access, etc), there was a relatively easier way to circumvent that…most likely find an engineer with access to the cluster and poach his/her credentials file.  Depending upon what you wanted to do, having this level access would be even better than having ‘root’ on the machine itself.

        Also interesting, is that this process sounds pretty similar to what ‘Law Enforcement’ might ask for if they served Amazon with a request for information on some AWS customer running ‘Scoundrels Bay’. Namely snapshot any EC2 instances and make a backup of the databases, and do this without them knowing about it.  Not saying that’s what happened, but if I were a Russian hacker trying to subvert democracy and really kick the bee hive in the process, I’d use the same tools as US Law Enforcement uses to subvert the system.  Possible moral of the story…any weakening of encryption and security to get at the ‘bad guys’ will eventually make it easier for hackers too.

  4. Greg says:

    orion it all depends. The DNC could have done 100 best practices related to security for these systems, but left 1 gap which nullified all the effort for the other 100 things. It would still sound mighty impressive to list the 100 things they did, but it really is besides the point if they missed something important.

    As for the questions on whether non-DNC personnel (GRU/etc) could have accessed the data after they “snapshotted” the servers? It all depends. But if I had to guess I’d say they were able to access them, yes. I think it’s likely that most of the DNC’s security efforts were built around preventing people from accessing these servers in the first place. And that once the servers were compromised there was little else in the way of protections/security. This is just my guess, under the assumption that if all the data on the servers was locked down super tightly, it would make it really annoying for regular DNC staffers to do their day job (analyze and query the data).

    • bmaz says:

      So, you are suggesting that if a political organization, with limited resources, does not DO EVERYTHING IMAGINABLE TO STOP MATRIX LEVEL HACKS, they are the ones at fault?? Is that what you are saying first time commenter? And, if so, explain yourself and why the entire political world need be held hostage by asshole hacker punks. Thanks, I will await your reply.

      • Greg says:

        No, I’m really not trying to suggest that at all.

        If something criminal happens, the criminal (GRU) is responsible. Not the DNC nor AWS.

        What I’m sensitive to is that reading this complaint, a lot of readers here could assume or start pointing fingers at AWS.  I see the word AWS again and again in the original post/complaint.  I’m trying to *add value* to the conversation by helping give my perspective on what it’s like to use a cloud service like AWS.

        I’d have made the same post if it was Linode, Heroku, Digital Ocean, Rackspace, or any other company in the same business as AWS.

        I assume you can see my email address (I’d prefer if it wasn’t public); feel free to email me if you really think I’m some kind of troll. I’m just trying to help.

        • orionATL says:

          greg –

          thank you for that response. though not a computer engineer type i know and fear this problem from efforts to protect my little home system:

          “orion it all depends. The DNC could have done 100 best practices related to security for these systems, but left 1 gap which nullified all the effort for the other 100 things. It would still sound mighty impressive to list the 100 things they did, but it really is besides the point if they missed something important.”

          that’s the devil of protecting concatenated data. a malfactor gets his nose under the tent anywhere along the perimeter and he’s in. i’m reading about security folks trying to work this problem out, but the old ways are entrenched because convenient (until the disaster, e.g., 500 million ?? customer records at mariott) and the current hardware design encourages it too.

        • Tech Support says:

          Interesting side note about the Marriott hack… the reporting was specifically that the Starwood data was compromised, and that it went back several years. In fact, it began before Marriott bought SPG.

          First that suggests that Marriott customers who have never stayed at a Starwood property were not impacted

          Secondly that implies that Marriott bought a compromised database and didn’t know it. I’m guessing SPG wasn’t aware either since if they failed to disclose the data breach during the due diligence portion of the sale, they’d be in deep kimchi right now. It was the disclosure of the data breach @ Yahoo that torpedoed their sale to… Verizon I think it was?

  5. AnotherKevin says:

    I just want to clarify, when you write “In other words, at least one of those snapshots was stolen after Stone suggested he would like better analytics data than what GRU had publicly released via HelloFL.” you mean that Stone made this suggestion on Sept. 9, while the snapshots were made between Sept. 5 and Sept. 22? Just wanted to be clear, thanks.

    If that’s the case, it would be very interesting to see the actual dates when the snapshots were taken, though I guess for legal purposes it only requires one to be in the necessary date range (post Sept. 9).

  6. Greg says:

    Honestly I think I was just excited that I could finally offer something to the conversation on Emptywheel, so I tried to write a thoughtful post (which I guess comes off as aggressive?).

    There is literally nothing as it relates to Mueller’s strategy or anything with the investigation where I can offer any value. But on this article, I thought that a lot of readers would think that:

    a) AWS itself got hacked (but it’s far from clear to me that this is the case)

    b) Even if AWS didn’t get hacked, that they are a major part of this story and might have somehow screwed the DNC.

    But based on my experience using AWS and other services to run my own web infrastructure, it seems much more mundane than that. I was happy that I could try to offer something intelligent up.

  7. WilliamOckham says:

    Going to jump in here and say that Greg and Bmaz are talking past each other. If you are, like Greg and me, AWS geek customers, this complaint seems weird in describing this as an AWS hack. We wouldn’t talk about it like that because “AWS hack” is a term of art that means something very different than what happened in this case. On the other hand, in this context, it was absolutely an “AWS hack”.

    Greg, the complaint is written by people who think of Tableau queries as computer engineering (which is totally reasonable, but people who write Tableau queries are just semi-sophisticated end users from my perspective). AWS security is ridiculously complicated (and I say that as someone who is an expert on Windows ACLs, much to my own chagrin). That the GRU only got to the test servers tells me that the DNC was doing better than 99% of the Fortune 500. Nobody can defend against a prolonged nation-state attack.

    Bmaz, cut Greg some slack. When somebody calls this an “AWS hack”, it sounds to him and me just like “collusion” does to you.

      • bmaz says:

        That is swell, when there is an actual explanation from “Greg”  that makes any sense as opposed to garbaging these threads at this blog without actual explanation, get back to me.

        • orionATL says:

          bmaz –

          the commenter greg, and several others here, have made comments that i find helpful in understanding better the dem hacks of 2016, and just as helpful, if not more, the current state of computer security. from the first to the last comment over several hours, this has been a very interesting discussion from my layman’s viewpoint.

          can you comfortably set yourself up as the sole arbiter at this site of who may give and who may receive info they consider helpful to greater understanding? because that is what you have been implicitly doing by dogging this discussion.

          how about laying off and letting others of us decide what to take and what to leave?

        • Scott says:

          Agreed. Every technical comment I’ve seen here seems perfectly accurate given how I know AWS to work. The first guy gets accused of being a shill or worse and RandomOPsGuy’s more detailed description gets a “Whut?”. I can appreciate they’re constantly bombarded by concern-trolls, but this seems like it spun out of control.

           

        • bmaz says:

          Yes, it did seem to spin out of control. For that I apologize. People consistently ask for plain language explanations of very complex legal concepts, and I endeavor to provide them when I can. Yesterday what is, apparently, a complex data argument was made on what was, otherwise, a rather pedestrian post on Stone and CFAA in an amended legal pleading.

          I asked for an explanation and, frankly, did not think I got it. I am still not sure I really understand the explanation, but that is okay. And then a bunch of first time commenters piled on. I guess that was fair. So be it. Thank you for your comment (and, yes, I mean that).

    • bmaz says:

      Yeah? Then explain. I try to do that for people that do not understand law, let “Greg” do so here and now for those that do not understand another specialty. I’ll be waiting.

      • NorskieFlamethrower says:

        “…explain yourself and why the entire political world be held hostage by asshole hacker punks…I’ll be waiting”

        Me too though I’m not holding my breath for anything intelligible on that or any other question.

      • RandomOPSGuy says:

        I hope I can help a little as I work in a heavily regulated and audited sector where we operate systems, both cloud and internally hosted.

        As mentioned before, the particular cloud service and software doesn’t really matter. In a corporation, you abstract these concepts to what’s generally referred to as “access controls” since an IT operations team might manage hundreds of applications on many different servers, cloud environments, etc. for a business.

        The described use of 2FA, IP restrictions, VPN’s, etc. are all very common, well-accepted ways to get to a server securely, but it’s the rights and roles which define what someone can do on that server and what data is visible. This leads to the notion of  “segregation of duties” (SoD) which is used to define who can access what/where/how. WilliamOckham above refers to the ridiculously complicated AWS security model – it’s the combination of these settings at AWS, and the software applications which are used that define roles adhering to SoD.

        For example, a user role for a particular application can only access specific production data using that application, a developer typically doesn’t have access production servers/data and a support role might only be able to administer the server itself (e.g. turn services on/off, reboot, etc.) but not run any applications or access business data. There are many, many ways to segregate roles/rights depending on the needs of the business. Auditors (the usual suspects – PWC, Deloitte, etc.) periodically review the overall “access controls” framework to verify that a corporation is managing their environment using industry best practices in a secure manner.

        Without knowing any details about the actual hack, it seems whomever got access to an account with a role that could access the test environment and copy that off to somewhere else. If the logging, audit trail etc. is sufficient, management would then be able to reconstruct what controls failed and develop a remediation plan.

        Obviously, the above is a very simplistic description and is necessarily much more complex in practice. While I can’t tell from the above exactly what happened, hopefully this gives a little background on the overall management process for these kinds of environments.

         

        • Greg says:

          When you’re an AWS customer, the “management console” where you configure all your settings is the equivalent of the cockpit to a 747. There are so many “knobs” and “switches” that it becomes overwhelming to try and configure all of it properly. It requires:

          a) your own technical personnel. people who are not “computer engineers” (or close to it) are not going to be capable of setting things up properly in AWS.

          b) a well thought-through security plan. it has to be an organizational priority, where a team of people sit down and make careful choices and blather on for hours and hours about their security choices.

          I think what @RandomOpsGuy is getting at is that all of this stuff is really hard, and it seems likely that the DNC team left a vulnerability somewhere. And then that vulnerability got exploited.

          This wouldn’t mean the DNC was grossly negligent, nor deserve scorn or finger-pointing. But when you set up your services on AWS, AWS doesn’t give you a how-to guide. You’re own your own and it’s REALLY hard.

        • pseudonymous in nc says:

          I pretty much agree with this. Because AWS is an accumulated hodge-podge of services, the access-control model is a hodge-podge. It’s basically what happens when people who understand operating-system ACLs think the same model can be extrapolated to organisations…

          I will say that the GRU indictment goes into a lot of detail about the attack vector on the DCCC and DNC networks and the tools used to sustain the attack, but only says that the GRU  “successfully gained access” to the AWS test servers.

        • alaura says:

          @bmaz and the @randomOPSGuy explaination:

          I think the complaint paragraph in Marcy’s post, beginning with this text, creates the impression that you can get from an email phishing link all the way to a “snapshot” of a virtual server on AWS.
          randomOPSGuys outlines many of the technical problems with that (…I was thinking along the same lines).

          “The DNC protected all of the data and code in its AWS servers by, among other things, restricting access to authorized users…”

          In other words, being an authorized user is no guarantee you can even get close to the access the hackers got. Access to the test severs is typically engineers-only, with fine-tuned access privileges around everthing they could touch, run, look at, etc. You could be allowed to run queries, but not edit them, for example.

          The complaint talks about general security but there’s a huge hole. It does not explain how the hackers got some of the necessary super-user privileges (see @randomOPSGuy).

          The hackers either had super-user privileges to do what they did, or they were able to hack the AWS platform to give themselves the super-user access once they got a little way in. Considering that I find it hard to believe the hackers somehow landed a systems engineer with the exact super-use privileges that would ALSO fall for a phising scheme, I think it’s got to be either an inside job or the hackers got enough access to the servers to exploit AWS system vulnerabilities to give themselves super-user privileges.

          AWS, Microsft, company XYS, they all have vulnerabilities.
          And we should go back to paper ballots =:)

        • Trip says:

          Any idea why Assange would suddenly be presenting a document of AWS servers? That seems to be a gigantic clue.

        • Trip says:

          The timing of it indicates trolling. As in he’s inviting/challenging more people to compromise AWS, or he’s telling everyone that whoever gave him the document drop already had that info.

          Note that, a week after DNC submitted its amended complaint on October 4, WikiLeaks released a proprietary AWS document showing the locations of all AWS’s servers around the world — something that is not all that newsworthy, but something that would be incredibly valuable for those trying to compromise AWS. That was one of its only releases since the crackdown on Assange has intensified.

        • alaura says:

          There’s no question they got serious access to stuff, sure. I was delving into the issue of HOW they got the access privileges, and how the complaint has a gaping hole in explaining how the hackers actually got as much as they did. Did they exploit AWS vulnabilities, or did someone inside help them or did a sys admin get drunk one night and shout out their super-user account credentials?

        • P J Evans says:

          I think that they aren’t about to give out the how of that hack – it’s revealing sources and methods, and that’s generally Not A Good Idea.

        • pseudonymous in nc says:

          The question is whether the security model for the servers was reflected in the security model for the AWS management console. And we don’t know that. I’d agree with the ops people here that the Occam’s Razor answer is that the GRU somehow got management console access, perhaps by spearphishing someone with console privileges limited to the test cluster or social-engineering a credentials reset. Everything else is at a higher level of hackery than the GRU indictment details.

          It’s the equivalent of stealing or duplicating credentials to the datacenter, gaining physical access to the servers and cloning a drive array.

      • J R in WV says:

        From my perspective, Greg did a fair job of explaining clearly what he meant. The fact that you don’t understand his explanation is not his fault. Chill out Bmaz, his explanation is clear and obvious to anyone who read it for content instead of with aggression.

        • Trip says:

          I’d say he’s pretty chilled out since the comment you’re responding to is more than 4 hours old and he pretty much dropped it.

          Just saying, let it go.

        • Tom S. says:

          Unless you’ve experienced it first hand, most do not have a feel for what it is like managing the comments stream of this or other sites dwelling on controversial, politically charged subject matter that also can be assumed to attract keen interest of government, domestic or foreign. The staff deserves the benefit of the doubt no matter how they call it because only they know what they are seeing, such as how many seemingly ( to us on this side of the curtain) distinct commenters are sourced from the same or associated IP #’ s. This is complicated by commenter use of VPN and other masking. Mods have tools and are suspicious until an interval of time and activity of a newbie provides data indicating otherwise, that the commenter likely is a friendly. The staff has seen things, they have accumulated a body of experience. The regulars are another matter. Some seem too clever by half. Follow the lead of staff reaction to newbies and the general atmosphere may grow more welcoming. BTW, Trip, read your comment in the other thread. I do need to work on my presentation, there is only one shot at making a first impression.

        • orionATL says:

          “The staff deserves the benefit of the doubt no matter how they call it because only they know what they are seeing, such as how many seemingly ( to us on this side of the curtain) distinct commenters are sourced from the same or associated IP #’ s.”

          NO!!! Absolutely Not!

          in an open discussion of a controversial subject the managers, the moderators, the bosses, the cops, the fbi, the corporation lawyers, the president’s men NEVER deserve the benefit of the doubt. when given that power freely, they will much too frequently abuse what was ceeded them to shut down, limit, or twist the discussion to serve their own needs.

          the words and ideas themselves are all that is needed to evaluate what has been said. intervening “authorities” are neither needed nor trustworthy.

        • P J Evans says:

          Dissing the lurking mods is never a good idea, and this site has very good – and not overly restrictive – moderation.

        • bmaz says:

          We  are NOT the cops, FBI, corporate lawyers or President’s men. We have an extremely permissive and open comments section when many other sites our age that have been around this long have gone to no comments or ridiculously limited ability to do so. There is some moderation, and there will continue to so be. This site is not Gab or Reddit. And, it is not going to be.

        • orionATL says:

          thou protests too much, dear bmaz. you know what i mean.

          that comment, by the way, was not aimed at you or your colleagues but at the few authoritarian souls commenting here whose stated first impulse was to trust the good intentions of authority (yes, even you :). that is “natsec” type thinking which is always an act of folly.

          “in an open discussion of a controversial subject” is the key phrase.

    • skillethead says:

      This isn’t really complicated.  Greg made an honest comment in which he showed that bmaz didn’t understand something.  bmaz cannot handle that.

      I’m about six months into following this site.  But this thread makes me think it’s better to spend my time elsewhere if people cannot simply admit they are wrong from time to time.

      • bmaz says:

        That is incorrect. I saw what looked like gibberish in service to an entity and asked for an explanation, and got more of the same. And then a lot of scolding.

        So be it, I will accept that and probably deserve it. The only one who came close to actually making sense of it was William Ockham, and I thank him for that. Thank you for your input.

        • NorskieFlamethrower says:

          Hang in there bmaz, the comment by Greg accomplished what I believe was his (or her) objective which was to hijack the thread from discussing the increased odds of a Stone CFAA indictment. You don’t owe anyone an apology.

        • cwradio says:

          BMAZ, I think you should cut back on the bacon. As the great American philosopher, Satchel Paige, once observed, meat “angrifies the blood”.

  8. bg says:

    I have been using MiniVAN for a few years now, for canvassing. The first time I was given the code for 2-stage verification was after July this year, I think. I did hear from a DNC presentation a few years back about the manner in which data on voters is collected. I was impressed, not that the data was collected but from where, like those shopping tags from big grocery stores that offer discounts for your use of the tags at the checkout. Not just on food but magazines and the like. It is all for sale, so the small discounts at the register are more valuable to the merchant than the discount as promoted. Of course your zip code and  on-line habits are part of the things you would imagine go into the DB. I know campaigns have to pay a high price to use MiniVAN, and I know there is currently a considerable fight with the DNC over who owns the data and who (state parties for example) have the rights to access all the data that canvassers are collecting at doors. Not that this is particularly relevant to the current discussion, but perhaps it is useful for those who are not familiar with the common usage as it applies to campaigns and the current back and forth over who should get access. As a further aside, volunteers like myself are allowed access to the “voter file” for organizing our neighborhoods, but there is very controlled proprietary access to that.

    • orionATL says:

      thank you. this is an interesting comment about minivan which was used in the recent georgia elections and i would bet elsewhere this year for the first time for some canvassing efforts (though it has been around for a while).

      as for security issues, i don’t know. if copped, the lists and turfs provide opportunity for vote suppression mischief i suppose.

  9. Jonathan says:

    Re AWS. I can’t see why anyone with sensitive data would trust an outside vendor with it whether it be AWS, IBM, or the man in the moon. It would seem that the secure way of doing things, would be to keep all the sensitive data on internal servers that are air gapped, in a physically secure location. The DNC certainly could afford the engineers it needs to provide data services inhouse, and that would have precluded a whole lot of heartache. I don’t care that outsourcing computer operations is the flavor of the month, that it looks to be cheap in the short run, or that lots of other businesses are doing it. As they used to say in the schoolyard: if everyone else jumps off a bridge, does that mean you should, too???

    The idea that AWS just scored a $10 billion sole source contract to manage Pentagon data is absolutely bonkers.

    I get that as a small business I might one day store data in encrypted form for disaster recovery at one of the services that does this. As a layman I just have to trust that encrypted data would be of no use to a hacker. The DNC, however, should be able to hire pros with no such issues. So should DoD for chrissake. Just why do taxpayers fund NSA, anyway?

    • P J Evans says:

      Air gaps only work if there’s no way to bypass the gap – air-gapped computers have been hacked. It’s not a good preventive measure.

      The company I worked at had different levels of access privileges – if you didn’t have an actual job need to access a particular database, of the many they had, you weren’t going to get in. And even if you needed in, you probably weren’t going to have anything beyond read-only, without really good reasons. (I had more privileges in one system than my lead person – but I’d had need, some years earlier, for those extra ones, and they hadn’t removed the access as they probably should have done.)

      • Jonathan says:

        Yes, obviously air gapping is not a cure all and of course air gapped machines have been hacked. But you do understand the spirit of my comment, I hope. My main point is that leaving data on a third party service… or online at all… is an invitation to trouble.

        • P J Evans says:

          There are many things that must be done online – or at least on a VPN. You can’t run most businesses without those connections. (If you have more than one location, you’ll need to connect them somehow, and sneakernet isn’t a good way to do it. Mail gets lost or delayed, even if it’s internal mail only. And having done sneakernet with just two machines on the same floor, it takes a lot of time.)

          ETA: not only that, but some applications don’t play well with some operating systems. Specialized software is going to have additional limits.

    • Greg says:

      @Jonathan it’s a good question. There are a lot of reasons but one answer is that the technical complexity of setting all of this up internally on your own hardware is extraordinary. It would entail re-inventing the wheel on so many things that the cost would be way, way higher than using 3rd party services. AWS and other service providers have gotten the operational aspects down to a science. They can employ teams of highly-paid specialists to create efficient ways to manage all of the hardware and software necessary. If not for open-source software and cloud-based services, organizations like the DNC would likely not even *attempt* to do some of the same projects. They would know they were in way over their heads.

      • orionATL says:

        pinc –

        my wife and i “air gap” our home system every night before bed by unplugging the modem from the copper wire. it may be futile, but it makes us feel better. at least it can cut down on uninvited and potentially troublesome 4am software updates. ☺😁

        • P J Evans says:

          I turn my computer off before I go to bed. (Exceptions in the past, occasionally, when I was downloading stuff that would require three or four days to completely arrive – and turning it off would mean starting over.) There’s nothing so urgent that it can’t wait until morning, when I turn it back on.

    • pseudonymous in nc says:

      It would seem that the secure way of doing things, would be to keep all the sensitive data on internal servers that are air gapped, in a physically secure location.

      The DNC had just had their entire network owned.

  10. Yogarhythms says:

    EW thank you for this thread about RS RtFkr and AWS,. Bmaz is on it. I’m not the tech savvy member of my family but knowing that GRU hackers exfiltrated DNC analytics firm AWS to other AWS surreptitious GRU accounts is a significant bread crumb in this story. Greg please show some honesty and restraint.

  11. Naomi says:

    Marcy didn’t dis AWS, she showed their world map had been published. When I read that, it’s sure a hack.
    Marcy showed we work in a target rich environment.
    Sorry for my typo in my signature. Naoi is usually Naomi

  12. oldoilfieldhand says:

    Marcy and bmaz are right…It’s about time someone called out the providers, by name. Can anyone point to an instance when any software developer, server or storage provider, including cloud services, offered anything except electronic versions of thoughts and prayers, and a promise to work diligently to correct “whatever anomalies” caused literally millions of customers adversely affected by unfortunate hacks, multiple problems, resulting from lost, misused or stolen data?
    Read the fine print. Wait, let me paraphrase the wording in plain language: “By agreeing to these terms of use, you fucked up, you trusted us and we are in no way responsible for anything; and whatever happens, we can’t be held liable for what we did in any way, except for the refund of the cost of the monthly service or initial software purchase price.”
    Would anyone buy a car or refrigerator with pre-condition liability acceptance disclosures even remotely resembling the gibberish related to electronic data storage?
    Also, too, the newer and improved version you will be required to purchase to protect your data against new attacks, which, by the way, contain the same blanket release of liability, will only offer limited protection from a related type of hack.
    Never trust anything to a server you wouldn’t want printed on the front page of the New York Times.

    • Michael says:

      “Never trust anything to a server you wouldn’t want printed on the front page of the New York Times.”

      Heh! I recognize that as an updated form of the old (1990’s) admonition, “Never put anything in an email that you wouldn’t want to read in tomorow’s [name-your-news-medium].”

      Back in 2001, my significant other, a clinical psychologist PhD, let it slip in conversation with me that she and her colleagues sometimes discussed client cases via in-the-clear email. I explained why that was very poor form, and warned her off, invoking the “Never put anything …” quotation. She installed PGP on her Win98 (yuck!) system, with my urging, and rocked on. (Which is why I pile – bmaz-strength! – derision on all claims that “PGP is too hard for non-geeks”.)

  13. greengiant says:

    A sign that EW has put another silver sword through the heart of some zombie rat fucker is the troll attempt to dust cloud the post comments. Like putting stuff out with one obvious falsehood just to generate more dust or back patting other trolls. There is more to the game than just PR control. Awhile ago one troll’s psycho babble led directly to Tom Barrack. For all that has been revealed multiple hacks of both AWS and the VPN could have been used. But who needs to know?
    The GRU having multiple contact persons within the Trump.org and operatives would not be a surprise. Can only hope Mueller’s net is broad enough to catch them all.

    • Trip says:

      Yeah, it’s so deep into the weeds now as to consume the real subject whole. And all from “Long time listeners, first time callers”. It’s like the JFK guy last night writing War and Peace size comments about something (but fuck if I know what).

      • Tech Support says:

        Or maybe EW finally posted something that let the computer geeks in the audience contribute instead of just the lawyers.

        • Trip says:

          As I read along, and after some ‘translations’ from others, I agree that that is the case. There have been people who try to hijack an article in the past. Not being well versed in some of the jargon, which I’ve admitted as much time and again, I believed that to be the case at first. I just wish that in addition to the technical aspects, someone had explained if any of that contradicts the essence of timing and Roger Stone. Maybe they have, but yours is the only comment I read so far this morning.

        • Trip says:

          I will add that one terrific thing about the lawyers who post here, is that they explain the law, usually in one step, in plain language for those who are not lawyers.

        • Tech Support says:

          That’s totally fair. Many in IT are, quite frankly, poor at communicating with other human beings. That’s part of the draw to IT in the first place.

          I’d like to believe I don’t have access to that excuse. As there will probably be more opportunities down the road I will be more careful to use accessible language and long-form stuff I might rely on shorthand to convey.

      • Tom S. says:

        Trip at 8:35 pm, I did not see your latest attempt to impugn my integrity. Do I have to include an image of my DL displaying an image of my face and my steer address to influence you to stop making your notion of an example of me? I provided my full name, but instead of taking some rather simple steps using the details easily found in a google search including that url I said I had edited the comments at, here you are, again. In my only other post in this thread, I reached out to you because in the other thread you had posted that your intent last night was not to run me off this website. Now, I have no reasonable choice other than to dismiss you as incoherent, at least in the entirety of the ways you have interacted with me directly, along with your drive by hete, Who are you? Are there instances on this site where you have shared your real name and tefetences and links to references of you with any continuity to anything you have posted here, as I have shared? You’re behavior is over the top.

        • Trip says:

          I still don’t know WTF you were driving at. And I have no intention of going on a wild goose chase of links from someone who won’t directly state a point. I looked back on that thread and noticed that you still hadn’t clarified anything. So I’m completely good with you dismissing me. I don’t care whether it is your real name or not, and I never questioned whether it was your real name because that’s not something that I care about.

  14. Trip says:

    There’s like a million comments and very few of them mention the ratfucker Stone. Thanks for your work, Marcy.

    • BobCon says:

      I am guessing that when Stone’s case is opened up, there is going to be a lot more evidence of communications with the people associated with Guccifer 2.0 that blows his cover story(s) out of the water. This whole dance really feels to me like Mueller has a ton of evidence and they’re working now to squeeze out the BS, although I realize it never makes sense to assume too much with either Mueller or Stone.

      And hey, I’m excited to see that Ayers is out of the picture as WH COS. They fired Kelly without a replacement — what a bunch of geniuses. My primary thought is that Ayers knows how bad things are going to get, although I have a thought I can’t quite squelch that there is lingering suspicion about a guy with such close ties to Pence. At any rate, I can’t imagine he’d do a good job shoring up the collapsing building, but he’d probably do a less bad job than many.

      And the replacements who are being floated — Whitaker! Mark Meadows! Mnuchin!

      Robert Lighthizer is probably a competent enough guy, but running USTR is wildly different from the White House. Mick Mulvaney is an weasel who would probably run Trump’s evil plans about as efficiently as possible, but I suspect he’s too smart to take the job.

      • Trip says:

        Yeah, I had the same thought as you with Ayers. He’s holding out for Pence, when Trump goes down. Whitaker! Mark Meadows! Mnuchin! All truly Trumpish choices. I’m surprised Scaramoochie isn’t on the list. No matter who it is, Trump won’t listen. He has no impulse control.

        • BobCon says:

          The latest is that Mnuchin’s not on the list, but who knows.

          Mulvaney is the one I worry about the most – he is about as ideological as Meadows, but a lot more effective at harnessing that to a large organization. He may be too smart to take the job, but he also has a huge ego and might think he can make it work for himself. And the press somehow refuses to see him as an ideologue.

        • BobCon says:

          And now Mulvaney and Lighthizer supposedly have walked away.

          They must think the ship won’t float for another year, even.

        • BobCon says:

          I would not underestimate his staying power, simply because there are significant hurdles to taking him out, and he has nowhere else to go.

          And I would not be surprised to see him take his aim off of Democrats and start humiliating some  Republicans on his way down. He may well surprise some of the GOP who think they can cross him for easy points in the post-Trump GOP.

        • Trip says:

          Laura Rozen thinks that maybe part of Cohen’s job wasn’t simply handing out payoffs, but issuing threats about exposure too.

        • BobCon says:

          I’m pretty sure that was true about Schneiderman, and Roger Stone was pretty clearly connected to bringing down Eliot Spitzer, with implications that he was playing Spitzer for a while before the story went public.

  15. Richieboy says:

    How about I just wade in right about now with an attempt at explaning and contextualizing the technology flapdoodle occurring above, the better to foster understanding and friendlier feelings amongst the disputants?

    Nah.

  16. Trip says:

    What about the firewall that went down between Democratic campaigns? NGP VAN in 2015, I think. Wasn’t that the same type of data that was accessible?

    Someone within another vendor might have the know how if a mole?

    • Trip says:

      And the one guy from Sanders’ campaign was fired for testing the breach. I’m not asserting that he has anything to do with it, just tossing it out there that there were known problems, even internally, never mind outside servers and whatnot. Unless that was actually a symptom of the larger breach.

  17. Alan says:

    The attacks on Greg were over the top and uncalled for. Greg, I find it regrettable that you were subject to that treatment, and if my apology would mean anything (in as much as it would be coming from someone who was not involved in attacking you), then I would offer it.

  18. Rusharuse says:

    What the fuck is a cloud?

    BTW, when WSJ reports-“Rupert seen chatting to Nancy” – you’ll know it’s game over!

    • Hops says:

      Historically, network diagrams have used a cloud symbol to represent an external collection of computers of unknown arrangement. The term has become popular as an adjective: cloud storage and cloud computing, wherein you rent time and space on computers in a data center accessed over the internet.

       

      • Ken Muldrew says:

        It’s like when you deposit money in the bank and they invest it in a bunch of crazy schemes. You tell people you’re storing your money in the cloud. When people ask, “what is the cloud?” you answer, “it’s the place that ATMs connect to”.

        Just go back upthread now and replace “data” with “money” and “data center” with “bank” and everything will make sense. Of course all large organizations maintain their own Scrooge McDuck style money bins for their financial security (not to mention the side benefits of allowing the executives to go swimming in the pile of loot during lunch).

        • Alan says:

          Not exactly, because when you take money out of the bank, they don’t give you back the money you put in, they give you money someone else put in, while in contrast, when you take data out of the cloud, they give you back your own data…  ok, scratch that–maybe you do have a point….

  19. HighDesertWizard says:

    I’ve been a professional enterprise database application software engineer for 35 years. I’m not a technology network, storage, or security expert but I work with these sorts of experts often. I’m familiar with the language they use and the way the talk about the various dimensions of control and administration going on. And I’m not an AWS expert.

    With those caveats a given, a few thoughts follow after I’ve reviewed every comment Greg made that are consistent with the feedback given by several others…

    — When I read the complaint, I had the same sense Greg did, that AWS was mentioned a lot when the issue probably had very little to do with AWS technology per se.
    — Lacking expertise, I didn’t speak up and I appreciated reading his insight.
    — He uses the same kind of language I’m familiar with.
    — I detect no hint of blaming or malicious intent on his part.

    My guess is that he appeared for the first time to comment on this post because he had insight and an opinion to share that he thought worthy of sharing. I think it was worthy of sharing.

    The analogy of a desktop computer at home being stolen in one of his posts is an appropriate, even great, analogy.

    It would be unfortunate, indeed, if other technical professionals were reluctant to share their knowledge and insight when it might be helpful because of how Greg has been harangued in these comments.

    • Greg says:

      Thanks @HighDesertWizard. I sometimes waste too much time pondering which analogy would be good, so it feels nice that you approve of this one!

  20. Callender says:

    This site is a jewel. It’s difficult to keep up with developments and MW is my hero because she and the others here do an excellent job of helping me do just that.

    This is one of the few sites I visit where the comments are as important to follow as the posts themselves. To that end, I commend bmaz and all the others who work to keep the comments from entropy. Far be it from me to criticize him, but he might be handling Greg a little roughly.

    Having said that, as someone else noted, Bmaz may have insight into the URL or know something about Greg we don’t. That’s my bet.

    Having said that, if king rat fucker RS winds up with a healthy dose of hard time it might turn me back to a “godfearin” man. RS is as crooked as a snake’s belly, and perhaps the best living example of this particular reptilian class since Lee Atwater.

    Lee tried to get into heaven by confessing on his death bed, because he knew there was a special place in hell for him otherwise. OT side note: we just lately learned that Gary Hart and “Monkey Business” was an Atwater rat fuck par excellence. There was no “there” there but Hart was ruined for one affair. How quaint that whole sad saga seems now in retrospect.

    The mind reels when one considers a counterfactual history posit that Atwater never developed brain cancer, and continued on in his reptilian career. We know where he would be today.

    Not sure Stoney understands how much trouble he’s in, because, like Trump, he’s been literally bull shitting his way out of every jam he’s ever been in since – well forever. Side note: His bull shit story on the scandal with that group sex ad in the porn mag sort of flew – like the Spruce Goose flew. That particular scandal is why he has to work from the shadows. That and it preserves his protective lizard slime.

    If the dude with the full-back Nixon tattoo goes down, I’ll die a much happier man.

    • NorskieFlamethrower says:

      Stone is so crooked they will need a corkscrew to get him into his coffin. I just hope they don’t burn ‘im up before he gets to hell.

  21. Alan says:

    I’ll share some insight that I don’t see mentioned here yet (although I haven’t read all of the comments).  The two factor authentication of the type mentioned in the DNC lawsuit sounds impressive, but it is close it worthless against an adversary such as the GRU.  Here’s why.  Let’s say one of the DNC computer admins has the cell phone number +1 301-867-5309.  First, the GRU steals the password that the admin uses to access the AWS management console.  The GRU might do that by using for example a zero-day exploit to compromise the admin’s personal computer.  This is “factor one”, and these are stolen all the time, which is the entire motivation for wanting “two factor authentication”.  Next, the GRU accesses a telephone switch located anywhere in the world, perhaps for example somewhere in Russia.  It can be any telephone switch for which they are able to gain physical access and install monitoring equipment, and surely for the GRU, there are many.  Next, the GRU causes the telephone switch to transmit a signal back to the USA telephone network that says “your customer with telephone # +1 301-867-5309 just connected to our telephone network”.  In other words, the GRU causes the telephone switch to send the same signal that it would send if the admin were traveling in Russia and turned on his cell phone.  Unfortunately, the is extremely easy to do because the telephone system is not secure and relies on a “gentlemans’ agreement” not to send false routing information, and there is little or nothing to prevent any switch operation from sending a message of this type.  Next, the GRU attempts to log into the AWS management console using the DNC admin’s password that they stole earlier.  The AWS system dutifully sends a short-term access code via txt msg to the admin’s cell phone.  This is the second factor.  The admin phone carrier however believes that the admin is traveling Russia and just connected to a phone network in Russia, so it dutifully forwards the txt msg from AWS to the phone switch in Russia.  The GRU intercepts this txt msg and now they have the second factor, the short term access code, and they type it in and viola, they have complete access to the DNC account on AWS, and they proceed to use the AWS management console to “snapshot” one of more servers containing the DNC’s data, and they transfer it to an account they control, as described in the DNC lawsuit.  The unfortunate fact is that this type of two factor authentication, which uses a cell phone to receive the second factor, is vulnerable to anyone who has access to a telephone switch in any country in the world, which basically means millions of people.  Please don’t take my words for it, google it and do your own research.  And FYI, if you have data that is important to you, do not use two factor authentication that depends on your cell phone, instead, invest in two factor or multi-factor authentication that uses a hardware key (preferable not a USB dongle either).  Here’s one reference (from the NIST to jumpstart your research).

    https://www.schneier.com/blog/archives/2016/08/nist_is_no_long.html

    • Alan says:

      sorry for the typos–the “Edit” button didn’t appear, which foiled my plan to submit the comment then clean it up.  But I’m sure you get the point.

    • Alan says:

      To summarize what I said above as succinctly as possible: the DNC was using a form of two-factor authentication (where the second factor was sent to a cell phone) that the US government (specifically, the NIST) has said DO NOT USE because it is not secure.  So let’s not be surprised that the GRU was able to break it.

    • Greg says:

      Great stuff @alan!  And also @TechSupport commented on your post far up above (it’s hard to track the various offshoots of the conversation).

      I went back to the complaint to try and get my head around some of the talk about two-factor authentication, and whether Stingray or some other kind of SMS intercept could be plausible here. My take is that the wording of the complaint is, at best, ambiguous. Obviously the purpose of the complaint is not to offer a comprehensive technical analysis of what happened, and I don’t know what more we could reasonably expect. But unfortunately without more comprehensive details spelled out in the complaint, some of it reads to me almost like hand-wavy gibberish.

      First, this:

      182. The DNC protected all of the data and code in its AWS servers by, among other things, restricting access to authorized users. To gain access to the AWS servers themselves, an authorized user had to take multiple steps. First, the authorized user would have to log onto a Virtual Private Network (VPN) using a unique username and password. Second, once the user entered a valid and password, the system would send a unique six-digit code (PIN) to the authorized user’s phone, and the user would have 30 seconds to type it into the computer system. This two-step process is commonly known as “two-factor authentication.”

      The above doesn’t make it quite clear what systems they’re talking about. Who operated this VPN? It might have been an AWS product, and it might not have been. Was the two-factor auth described here related to the process for signing in to the VPN? It sounds like it. But first of all, that would mean this has nothing to do with the AWS management console, where servers and resources would be provisioned, nor nothing to do with AWS security policies or user management. If an unauthorized user gained access to the resources behind this VPN, it’s unclear what it would allow them to do and how it would relate to snapshotting a test/dev server on AWS.

      Second, the description of how this two-factor auth works does not add up to me. Most 2FA that I use is *app-based*, not *SMS-based*. And it is *app-based* 2FA where I’m accustomed to a 30 second countdown before any given 6-digit code expires. If the codes were sent via SMS, I don’t think a 30-second expiration would be plausible. The codes would often expire before being received onto the phone, due to delays in the SMS being sent and then received. For comparison, the only *SMS-based* 2FA that I personally use is from Bank of America because they give me no other option. And in Bank of America’s case, they give me a full 10 minutes before the code expires.

      All this leads me to think that this was actually app-based 2FA, and the complaint is *incorrectly* describing how “the system sends a code to the user’s phone.” If it was app-based, no system is sending a code to the user’s device. The codes are generated on the user’s device**, and phone/SMS transmission is not involved. And if that’s the case, Stingray would not have been useful in any way, right? And, if this is where the GRU were able to find an exploit and gain access, it further seems like it doesn’t really have anything to do with an AWS account or AWS itself being compromised.

      Next, this:

      183. Authorized users would also employ a two-factor authentication system to access Tableau visualizations. First, they would log into a Google account with a unique username and password, and then they would enter a pin sent to their cell phones.

      Just a minor point that the above also seems a bit… weird. What did a user’s Google account have to do with their Tableau access?

      a) It sounds like they are describing how DNC users authenticated into Google (probably for email, calendar, etc). But I’m unsure, as a (novice) former Tableau Desktop and Tableau Server user, how this would be connected to a Google account.

      b) In my experience with Google 2FA, the user can choose their 2FA method. So unless the administrator of the DNC Google organization *imposed* *SMS-based* 2FA on their users, it would seem unlikely to me that it was actually SMS-based at all. Instead would be more likely that a given user would install Google’s own authenticator app and use that.

      Next:

      184. Finally, the DNC’s AWS servers were protected with firewalls and cybersecurity best practices, including: (a) limiting the IP addresses and ports with which users could access servers; (b) auditing user account activities; and (c) monitoring authentication and access attempts.

      So *this* above actually to me sounds like the simplest attack vector where DNC could have left themselves exposed and for the GRU to have snapshotted/compromised a test server. If there was *any* mistake or oversight made in these “cybersecurity best practices” of closing down ports and IP ranges, on *any single server*, it would have been perfectly plausible for the GRU to have connected to this server, possibly even with root privileges, and SCP’d everything out to their own environment.

      I guess in summary, I don’t really find the complaint to be written carefully and comprehensively enough from a technical point of view for any of us to really be sure about a lot of these things.***

      **I don’t know the details of exactly how authenticator apps from Google and Authy remain synchronized with the server, but my assumption was it’s a mathematical sequence that kicks off between the server and the client, and once that sequence starts, there no longer needs to be any communication back and forth. Maybe that’s totally wrong.

      ***I am so, so far away from being a security expert, that perhaps I’m making some bad assumptions here, based on the technical knowledge and experience I’ve got.

      • Alan says:

        yes and no. An app running on the phone would be better than SMS if and only if the app were “mated” or “paired” to the login server so that the two were communicating on a secure (encrypted and bi-directionally authenticated) channel.  If not, then the app would still be vulnerable to a man-in-the-middle attack and as easy to compromise as SMS.

        If the app was mated to the login server, then the second factor could have still been stolen from the phone by compromising (installing spyware on) the phone, or by active attacks such as inducing the computer admin to re-mate the phone to the login server (for example by using packet manipulation to prevent a successful login), and then intercepting and stealing the mating parameters and loading them into a device operated by the attacker, or by causing the attacker’s device to mate to the login server instead of the admin’s phone.

        An app on the phone can make it harder than simple SMS, but it would not be insurmountable to a sophisticated attacker.  A dedicated hardware device is better than a phone because it is harder to compromise, but active attacks as described above still work against dedicated hardware devices.

        • Alan says:

          P.S. Google 2-Step Verification, as of today:

          “a code will be sent to your phone via text, voice call, or our mobile app.”

          https://www.google.com/landing/2step/#tab=how-it-works

          So if the DNC was using google 2FA, and if it works that same as google describes today, it depends on if they had it configured to use voice, txt or the mobile app, and if they were using the mobile app, it depends on if the mobile app was “mated” to google servers, and even if it were, they could have still stolen the second factor using spyware or an active attack.

        • Alan says:

          PPS, I completely agree with the premise of your original post–the likelihood that the GRU compromised the two-factor authentication used by a single DNC admin is orders of magnitude more likely than the GRU compromising the AWS infrastructure.

        • Greg says:

          Cool but I think maybe that “how it works” language has been dumbed down for marketing purposes?

          I don’t think, if you are using Google’s authenticator app, that the code is really being “sent” to the device. I think the codes are independently generated on the device and the server. I found this article, not sure how reliable it is:

          https://medium.com/@tilaklodha/google-authenticator-and-how-it-works-2933a4ece8c2

          But this says there is an initial secure key that is created upon the setting up the 2FA. At that point, this secure key is combined with a current timestamp in order to generate a one-time passcode. So long as the server and the device think it’s roughly the same time and day, the the passcodes will be independently generated without anything being transmitted between them, and the codes will still match.

          Anyway, even if it *was* transmitted/sent to the device, I’m guessing it would be a heck of a lot harder to use a technology like Stingray to try and extract that data (which would only be good for 30 seconds), as compared to if the code were sent via SMS?

          So I’m just wondering how likely it is that this is what happened in terms of the analytics data breach. As compared to my alternate theory that they left a port or root user access available on a single box (hosted on AWS), and the GRU found it.

        • Alan says:

          The time-based access codes could still be compromised by spyware on the phone, or by an active attack (preventing successful login until the admin attempts to reset the app on the phone, and then intercepting the parameter exchanged during the reset).

          Root access to a single box is not consistent with the claim that the GRU snaphot’ted the servers and then transferred the snapshots to their own account.  This is me is consistent with something that an attacker would do on the AWS management console.

        • Greg says:

          But what if we consider that the word “snapshot” is being used for layperson purposes, and not what technically happened…?

        • Alan says:

          You might catch an attorney using the word “snaphot” to refer to a photograph, but this use came straight from the engineers, and it most likely means exactly what you think it means (a snapshot of an EBS volume)

        • Greg says:

          Anyway, you might be right, and it’s probably not worth my continuing to “what if” … just enjoying the conversation and thinking about it at this point. Thanks!

        • Alan says:

          PPS, even if someone is using google 2FA with the mobile app, it might be possible for an attacker to reconfigure the authentication to use SMS (hack into the user’s google account? social engineering?)  This is a common attack vector: if a system supports multiple protocols, try to reconfigure it to use the least secure, or fall back into using the least secure. A secure system never allows a less secure option, and because google 2FA does (it supports SMS), this leads me to question the soundness of their engineering.

        • Greg says:

          Alright but let’s say they *did* compromise an admin user’s authentication (into either the VPN, a Google account, or their AWS account).  I’m having a hard time seeing how the other facts presented in the complaint line up with this. It seems like 1 or maybe a handful of test/dev servers on AWS were compromised and then snapshotted. It’s so specific, that it makes me instead wonder if this 1 or handful of test servers was simply left exposed in a dumb, accidental way. GRU is sniffing around, finds they can SSH into these boxes (but no other boxes) and “snapshots” all of it. Thoughts?

        • Alan says:

          The purpose of the complaint is to describe the things of value and how they were wrongly compromised, in language that can be understood by a judge who has no technical background.  So that’s why it’s oddly specific–the complaint is describing two locations that held goodies, what those goodies were, how they were valuable, and how they were compromised.  The attackers got the keys to the DNC house, and while I’m sure the DNC has many more boxes sitting in the basement, only the jewelry boxes are described in the complaint.

        • Greg says:

          Very true. I was just thinking earlier this morning (and hence my long post) that other things in the complaint read so strangely and possibly inconsistent to me, that I’m unsure what in it really can really be trusted, as far as what technically happened.

        • Alan says:

          The complaint is the attorneys’ interpretation of what the engineers told them, and some of probably did get lost in translation…

        • Greg says:

          Yeah. Kind of ties into my thinking, maybe they heard the word “snapshot” at some point, then just fed it back into the complaint because it sounded cool, leading to a lot of confusion.

  22. Alan says:

    To summarize what I said above as succinctly as possible: the DNC was using a form of two-factor authentication (where the second factor was sent to a cell phone) that the US government (specifically, the NIST) has said DO NOT USE because it is not secure. So let’s not be surprised that the GRU was able to break it.

    • P J Evans says:

      And you get someone like me who uses the cellphone for outgoing calls only, not for text, and generally the phone is at home and in need of charging. If they go two-factor on me, they have to use email.

      • pseudonymous in nc says:

        If your email gets popped, though…

        One of the things that Maciej Ceglowski did as part of his security training for midterm candidates was to hand out YubiKeys for 2FA on email accounts and social media. For the moment, token-based auth isn’t as vulnerable.

  23. fikshun says:

    While the nature and the point of entry of the hack are important, what was hacked seems much more relevant.  This wasn’t just John Podesta’s emails that were taken.  They got a very deep level understanding of the DNC’s analytics, mode of thinking (via the spitball tests), and strategy.  The extent of this hack reads more serious than Watergate.  If it can be proved that Roger Stone was sending smoke signals to direct Guccifer 2.0 to go after different assets, that’s bad news for Cap’n Ratfucker, but that almost seems beside the point.

  24. HighDesertWizard says:

    Alan… Great info… Good on you for highlighting the change in NIST Guidelines about 2 factor authentication…

    I followed the link you provided and made some additional queries. I’m no expert, but it was easy to validate that a change in NIST 2FA guidance took place.

    One thing you didn’t mention and is worth EmptyWheel considering–because she’s got the time line in her head. Thanks EW…

    –> the date on the link you’ve provided is Aug 2016!

    I didn’t have time to investigate the actual guidance change date, but couldn’t find the date in a brief search.

    An interesting question to ask…

    Did the NIST guidance change take place after, and maybe even because of, discoveries made by the FBI Russia probe it had recently begun?

    Again, thanks for sharing this info.

    • Alan says:

      The insecurity of using a cell phone as the second factor was known to experts and hackers long before the NIST changed its guidance.  I would be skeptical that the Russia election interference investigation was a major factor or even one of the factors, but that would be a possibility.  There are probably people in this world that can answer that question, but I’m not one of them.

  25. Greg says:

    Thanks, those that had encouraging comments. They made me feel less self-conscious that maybe I was really out of line somehow. No further explanation and no apology needed from bmaz. People come here for various manipulative reasons, and many times I’ve seen bmaz correctly put them in their place. I just didn’t expect that I was going to get flagged into the same bucket, and when it happened I wasn’t really sure what to do. Every further comment I made ran the risk of digging myself further into a ditch!
    I run my own website (bmaz can likely see which one, from my email address, and my preference is to keep it private and out of the conversation), and from that experience I know how hard it is to moderate a discussion forum. No ill-will towards he and the others that run (and write for) this site that I enjoy so much.

    • bmaz says:

      Hi Greg – You may not be seeking an apology, but I will give you one anyway. It did indeed  strike me that there was something ulterior, your phrase manipulative reason is also apt, going on. I was clearly wrong, and for that apologize. As I acknowledged to Scott above, it seemed to spiral out of control. Your participation and comments here are always welcome, and I hope you continue to join in.

      • Greg says:

        Appreciate it – we’re all good. At face value my post looks like it could have been written by a damage control PR/legal operative from Amazon.

        Never intended for the thread to get hijacked so far away from the main substance of the original post!

  26. Philip S. Webster says:

    I read all of these posts and first was siding against bmaz because I have an interest in some of this shit. BUT, none of it is germane to emptywheel arguments or explanations about what this place is about. It does seem like trolling as it is so arcane to most.
    Keep the pedal to the metal bmaz.

    • orionATL says:

      read more posts here before you embrace your conclusions too strongly.

      comments following ew’s posts very frequently do not follow that post :)

      it is common here for a thread to develop on its as this one did. these discussions can be very rich and valuable, as this one was in my view.

      conversely, discussions that follow the topic of a post can seem tedious and empty as in the frequent “guessing game” style of commentary – why did manafort/mueller/giuliani/trump do or not do such-and-such – where dozens of disconnected guesses mill around.

  27. Jonathan says:

    @Greg I get that reinventing the wheel in computer land is a very fraught endeavor. Just from the sort of small scale computer projects I did in the pre-internet Stone Age, to work my way through grad school. And yet… in the end it is the customer’s responsibility to secure its own data. The names of major corporations and key government agencies that have been hacked, is pretty much the same list of … major corporations and key government agencies. If you are running one of these entities, how do you take responsibility for a major hack not to happen on your watch?? Not just, “we did the right things,” but to keep the data actually safe?

    I suspect that if you need to get something done quickly, maybe hiring a team from a contractor like AWS or IBM, to work on premises but NOT in the cloud, would be a start? I take your point about expediency and getting the job done. And maybe using a VPN to communicate with said entities’ remote locations. On the other hand, it doesn’t seem like ANYONE has been able to safeguard their key information once it is exposed to the Web. (And yes there are other sorts of attacks, but it would seem that common sense tells you to eliminate Web exposure if you are serious about data protection.)

  28. General Sternwood says:

    Okay, here’s my funny idea related to Stone’s former partner Manafort, who was in touch with a “senior administration official”. I was reading around trying to figure out who he might still be communicating with, and saw this by Philip Rucker and Matea Gold (Washington Post, 7/16/16):

    In recent weeks, as Trump’s vice-presidential selection process kicked into gear, Ayers developed close ties with Manafort and earned his confidence. Ayers has decided to follow Manafort’s lead in working for the campaign for no pay, while the rest of Pence’s team will be paid staff members. Serving as a volunteer helped Manafort gain Trump’s trust and respect this spring when he fought internally with other senior aides.

    I have a vague memory that Ayers was also rumored to be the “senior administration official” behind the NYT oped. It crossed my mind that Mueller might have been trolling the White House by using the latest Manafort filing to let the administration know that he had intercepts of the presumptive COS secretly communicating with Manafort. I realize that “senior administration official” can be any one of hundreds of people. But I like to think of Mueller in this light, half Lone Ranger and half Sadistic Mastermind.

    • Tom says:

      I saw an old clip of Omarosa on YouTube the other day in which she said she thought Nick Ayers might be the anonymous member of the Resistance in the WH behind the NYT editorial.

  29. Michael Keenan says:

    So at risk of getting an ass chew by the Bmaz what of Binneys claim of a leak over a hack given the technical feasibility of one by memory stick over a straight internet raid? It is kind of hard to ignore this guy given his standing. What saith the Tech Support guy?

  30. Eureka says:

    On September 20, 2016, CrowdStrike’s monitoring service discovered that unauthorized users had breached DNC AWS servers that contained testing clusters. Further forensic analysis showed that the unauthorized users had stolen the contents of these DNC AWS servers by taking snapshots of the virtual servers, and had moved those replicas to other AWS accounts they controlled. The GRU stole multiple snapshots of these servers between September 5, 2016 and September 22, 2016. The U.S. later concluded that this cyberattack had been executed by the GRU as part of its broader campaign to damage to the Democratic party. The GRU could have derived significant economic value from the theft of the DNC’s data by, among other possibilities, selling the data to the highest bidder.

    In light of the expanded dates (and, well, history), it is interesting that Stone tweeted twice, ca. 9-20-16 (I didn’t convert the timestamps):

     

    Sep 20, 2016 12:40:02 AM San Antonio @realdonaldTrump HQ broken into- sensitive campaign data stolen !#HillaryThugs @RealAlexJones @infowars @PrisonPlanetTV [TweetDeck]

    Sep 20, 2016 12:35:23 AM San Antonio @realDonaldTrump HQ burglarized and sensitive campaign information stolen #HillarysThugs @StoneColdTruth #Tx [TweetDeck]

Comments are closed.