The Other Servers and Laptops FBI Never Investigated: VR Systems and North Carolina Polling Books

Ron Wyden had a lot to say in his minority views to the SSCI Report on election security released yesterday, mostly arguing that there need to be national standards and assistance and that no one can make any conclusions about the effects of Russia’s efforts in 2016 because no one collected the data to make such conclusions.

But there’s one line in his section raising questions about the 2016 conclusions I find particularly interesting, pertaining to VR Systems (which he doesn’t name).

Assessments about Russian attacks on the administration of elections are also complicated by newly public information about the infiltration of an election technology company.

Since the Mueller Report came out, Wyden has been trying to chase down this reference in the report to the VR Systems hack.

Unit 74455 also sent spear-phishing emails to public officials involved in election administration and personnel a~ involved in voting technology. In August 2016, GRU officers targeted employees of [redacted; VR Systems], a voting technology company that developed software used by numerous U.S. counties to manage voter rolls, and installed malware on the company network.

In May, he sent a letter to VR Systems President Mindy Perkins, asking how the company could claim, in March 2018, that it had not experienced a security breach when the report said it had been infected with malware in August 2016. In response, the company told Wyden (according to a letter he and Amy Klobuchar sent FBI Director Chris Wray) that they had alerted the FBI that they found suspicious IPs in their logs in real time, but that FBI had never explained the significance of that.

In a May 16, 2019, letter to Senator Wyden, VR Systems described how it participated in an August 2016 conference call with law enforcement. Participants in that call were apparently asked by the FBI to “be on the lookout for certain suspicious IP addresses.” According to VR Systems, the company examined its website logs, “found that several of the IP addresses had, in fact, visited our website” and as a result, the company “notified the FBI as we had been directed to do.” VR Systems indicates they did not know that these IP addresses were part of a larger pattern until 2017, which suggests the FBI may not have followed up with VR Systems in 2016 about the nature of the threat they faced.

The implication from Wyden’s letters is that VR Systems only hired FireEye to conduct an assessment of what happened after Reality Winner leaked an NSA document making it clear they had been targeted by GRU in 2017. [Update: Kim Zetter actually reported this here.]

In their June 12 letter, Wyden and Klobuchar asked Wray whether the FBI followed up on VR Systems’ report.

  1. What steps, if any, did the FBI take to examine VR Systems’ servers for evidence of a successful cyber breach after the company alerted the FBI, in August of 2016, to the presence of suspicious IP addresses in its website logs? If the FBI did not examine VR Systems’ servers or request access to those servers, please explain why.
  2. Several months after VR Systems first contacted the FBI, electronic pollbooks made by the company malfunctioned during the November 8 general election in Durham County, North Carolina. In the two and a half years since that incident in Durham County, has the FBI requested access to the pollbooks that malfunctioned, and the computers used to configure them, in order to examine them for evidence of hacking? If not, please explain why.
  3. VR Systems contracted FireEye to perform a forensic examination of its systems in the summer of 2017. Has the FBI reviewed FireEye’s conclusions? If so, what were its key findings?

It’s unclear how Wray answered (or didn’t). But just before Wyden sent this letter, the WaPo reported that no one had yet conducted a forensic examination of the laptops used in the VR Systems polling books in North Carolina. After Democrats took over control, they finally persisted in getting DHS to agree to check the laptops.

On Tuesday, the Department of Homeland Security told The Washington Post it will conduct a forensic analysis of the laptops used in Durham County elections in 2016. Lawson said North Carolina first asked the department to conduct such a review more than 18 months ago, though he added that DHS has generally been a “good partner” on election security.

“We appreciate the Department of Homeland Security’s willingness to make this a priority so the lingering questions from 2016 can be addressed in advance of 2020,” said Karen Brinson Bell, the newly appointed executive director of the State Board of Elections.

After the election, Durham County hired a firm called Protus3 to dig into what happened. The security consultant said it appeared the problems were caused by user error but ended its 12-page report with a list of recommendations that included examining computers in a lab setting and interviewing more election workers.

Durham County elections director Derek Bowens said he is comfortable with the report’s conclusions. Even so, in 2017, the county switched to electronic poll books created by the state. Bowens said in an interview that the state’s software would save money and is, in his view, better.

But for North Carolina officials, concerns resurfaced in June 2017 when the website Intercept posted a leaked National Security Agency report referencing “cyber espionage operations against a . . . U.S. company in August 2016.” The NSA report said that “it was likely that at least one account was compromised.”

VR Systems soon acknowledged that hackers had targeted the company but insisted that its network had not been breached.

North Carolina officials weren’t so sure.

“This was the first leak that indicated anything like a nation-state actor targeting a voting systems vendor,” Lawson said.

The state elections board soon launched its own investigation, seizing 40 laptops from Durham in July. And it suspended the certification that allowed more than 20 North Carolina counties to use VR Systems’ poll books during elections, an action that would later land in court. “Over the past few months there has been a considerable change in the election security landscape and the level of scrutiny we receive,” the board wrote in a letter explaining its decision to VR Systems.

No one working for the board had the technical expertise to do a forensic examination of the machines for signs of intrusion. Staffers asked DHS for technical help but did not get a substantive answer for a year and a half, Lawson said.

As noted, FireEye appears to have done an assessment at VR Systems itself in the wake of the Winner disclosure. The WaPo reports that FireEye declared VR Systems hadn’t been hacked, but wouldn’t share any information with Wyden or–apparently–DHS.

VR Systems said a cybersecurity firm it hired to review its computer network in 2017 found no evidence of a hack. A subsequent review by DHS also found no issues, the company said. VR Systems declined to give Wyden documentation of those reviews, citing the need to protect proprietary information.

Wyden in a statement to The Post accused VR Systems of “stonewalling congressional oversight.”

A senior U.S. official confirmed DHS’s review of VR Systems’s network to The Post and noted that by the time agency investigators arrived, a commercial vendor had already “swept” the networks. “I can’t tell you what happened before the commercial vendor came in there,” the official said, speaking on the condition of anonymity to discuss a sensitive matter.

The same day as the WaPo report, Kim Zetter reported that VR Systems used remote updates for their software, opening up a possible point of compromise for hackers.

For two years, GRU hack denialists have thought it was the most important thing that the DNC provided FBI Crowdstrike’s forensic images of the hacked laptops, rather than providing the servers themselves.

But that step has, apparently, not been done yet with VR Systems. And the laptops that failed on election day are only now being forensically examined.  Which is why, I presume, that Wyden believes it’s premature to claim no vote totals were affected on election day 2016.

40 replies
  1. Ancient Mike says:

    So, to be clear, is anyone actually TRYING to find out whether any vote totals were altered in 2016?

  2. P J Evans says:

    It sounds like DHS wasn’t doing their job. Maybe they thought that it wasn’t theirs, as elections are run at state and local levels, but when there’s foreign interference of any kind, even just probes, they really need to look at it.

    • harpie says:

      On August 15, 2016 there was a conference call between election officials in each state and the DHS:

      [p47] (U) In an August 15, 2016, conference call with state election officials, then-Secretary Johnson told states, “we’re in a sort of a heightened state of alertness; it behooves everyone to do everything you can for your own cybersecurity leading up to the election.” He also said that there was “no specific or credible threat known around the election system itself. I do not recall—I don’t think, but I do not recall, that we knew about [State 4] and Illinois at that point.322

      The Committee notes that this call was two months after State 4’s system was breached, and more than a month after Illinois was breached and the state shut down its systems to contain the problem. During this call, Secretary Johnson also broached the idea of designating election systems as critical infrastructure.

      (U) A number of state officials reacted negatively to the call. Secretary Johnson said he was “surprised/disappointed that there was a certain level of pushback from at least those who spoke up…The pushback was: This is our—I’m paraphrasing here: This is our responsibility and there should not be a federal takeover of the election system.” 323 ] […]

      • harpie says:

        [p48] (U) States also reported that the call did not go well. Several states told the Committee that the idea of a critical infrastructure designation surprised them and came without context of a particular threat.

        Some state officials also did not understand what a critical infrastructure designation meant, in practical terms, and whether it would give the federal government the power to run elections.

        DHS also did not anticipate a certain level of suspicion from the states toward the federal government. As a State 17 official told the Committee, “when someone says ‘we’re from the government and we’re here to help,’ it’s generally not a good thing.”326

        • Herringbone says:

          “When someone says ‘we’re from the government and we’re here to help,’ it’s generally not a good thing . . .”

          . . . said a state government official.

          So when state government IT shows up to get this person’s government computer up and running so this person can do government work while getting paid by the government, does this person ever pause to reflect on their membership in the deep state?

      • harpie says:

        The first recommendation of the Committee report:

        [Pg.54] IX. (U) RECOMMENDATIONS
        1. (U) Reinforce States’ Primacy in Running Elections*
        (U) States should remain firmly in the lead on running elections, and the federal government should ensure they receive the necessary resources and information.

        • harpie says:

          From page 1 of Ron Wyden’s minority views:

          [p1] (U) The Committee report describes Russian attacks on U.S. election infrastructure in 2016 and lays out many of the serious vulnerabilities that exist to this day. These vulnerabilities pose a direct and urgent threat to American democracy which demands immediate congressional action.

          The defense of U.S. national security against a highly sophisticated foreign government cannot be left to state and county officials. For that reason, I cannot support a report whose top recommendation is to “reinforce [ ] state’s primacy in running elections.” […]

        • P J Evans says:

          I’d be willing to let the states have primacy in *running* elections, but we need federal standards for voting system security, from voter registration to ballot counting, and those standards need to be non-partisan and fair for *all* the voters, not just white male conservatives.

      • harpie says:

        So, a week before the August 15, 2016 meeting mentioned above:

        […] In early September 2016, President Obama had dispatched three senior U.S. officials including DHS Secretary Jeh Johnson, Homeland Security Adviser Lisa Monaco, and FBI Director James Comey to brief the “Gang of Twelve,” a group that includes the Gang of Eight plus the chairs and ranking members of the committees on homeland security.

        The White House wanted the congressional leaders to agree to “a bipartisan statement urging state and local officials to take federal help in protecting their voting-registration and balloting machines from Russian cyber-intrusions,” the Washington Post reported.

        McConnell nixed the idea and remained steadfast despite Paul Ryan’s effort to persuade the Senate Majority Leader to change his mind.

        Yet even McConnell’s stance—declining to issue a joint public statement—was far shy of Burr’s tack of making public statements inconsistent with the intelligence information. […]

        At this time, SSCI Chairman Richard Burr, who’s committee wrote this report, was in a tight race to keep his Senate seat, and was Trump’s campaign adviser on foreign policy. He formally joined the Trump campaign on 10/7/16.

        • harpie says:

          I would bet Kentucky and North Carolina were two of the states who did NOT want the federal government to help secure their voting systems.

      • harpie says:

        Also, I want to add this article and tweet thread from Kevin Collier:
        2:09 PM – 26 Jul 2019

        My followup from the SSCI report: Florida officials struggled to manage their cybersecurity and failed to heed multiple warnings as Russian hackers attacked: [link 7/26/19 to CNN article]
        [thread, continued]:
        SSCI gives the clearest depiction yet from something I’ve long heard grumblings about: just how hard it was to get some local officials to act. DHS has spent years now on a charm offensive, trying to get counties to use their cybersecurity services, & wouldn’t say it publicly […]

        • harpie says:

          And, remember Kemp in Georgia.

          9:03 AM – 26 Jul 2019

          “Georgia election officials repeatedly and intentionally destroyed evidence that could show unauthorized access to state election infrastructure and potential manipulation of election results, according to a federal court filing.”
          Jul 3, 2017: suit filed against B Kemp challenging corruptible voting system in aftermath of GA06 Special
          Jul 7, 2017: election officials wipe election server
          Aug 9, 2017: case referred to federal court
          Aug 9, 2017: election officials use magnets to manually destroy backups

  3. Herringbone says:

    Just by way of background, after 2004 there was a big push in North Carolina to revamp voting systems. Probably because of the convenience factor, most of the state’s urban counties continue to use push-button voting systems, but the vast majority of counties use optical scan. There are also random hand audit requirements in place (though I don’t know that these ever get done).

    Also, though, are VR Systems just about voter rolls? If so, how would that affect vote totals? Or does Wyden mean that votes might have been suppressed by deleting voters from the rolls?

  4. Marinela says:

    If votes were changed, this kind of compromat would explain why Trump is acting as Putin’s puppet.

  5. Blueride27 says:

    Realistically, what are the chances of voting Mitch McConnell out? Unless there is a blockbuster report, detailing every single step the Russians took to change votes. I don’t see the public at large listening to this. To me, this is huge. To the general public, too many big words and if it doesn’t talk about my state, why should I care?

      • bmaz says:

        That is not going to happen this election cycle. Maybe 2022. 2020, just is not going to happen.

    • Marinela says:

      If the Senate is still under GOP control after 2020 (likely), even if WH is democrat, I suspect Senate is going to impeach the next democrat President on fake charges. Can the Senate impeach without House starting the impeachment proceeding, or it must be started in the House? If so, if the House and Senate is GOP controlled, President is democrat, then I am afraid the GOP would impeach the President without a good reason. So could this be some of the Nancy calculations? She needs to make sure the House stays under democratic control.

      • bmaz says:

        No. Impeachment, whether a mere inquiry, as is needed now, or formal Articles of Impeachment voted out for trial in the Senate (which are not needed yet), is a creature that can only come from the House.

  6. earlofhuntingdon says:

    Safer and more accurate elections require uniform standards for holding them, communicating about them, and auditing them. In exchange, the feds will have to pony up some of the cost of making elections safer.

    But federal legislation would open a can of worms concerning how badly states typically contract for election machinery and services. For one thing, the audit and security provisions are terrible. The normal excuse is that the software and systems are “proprietary” and thus beyond both the technical knowledge of local administrators, and the reach of state disclosure and public audit rules.

    That leaves the public having paid for an expensive pig in a poke about the most important thing governments do: hold elections. But the the federal government has enormous legal and practical leverage to fix that by mandating uniform provisions.

    It won’t happen until there’s a Democrat in the White House. Republicans hate the idea – they don’t think it helps them win – and businesses do to. It would set a terrible precedent to require better contracts with the private sector, especially where accountability, public access, and public disclosure is the chief concern.

    • P J Evans says:

      One of my friends says that voting machines would be a lot safer if they were handled by Nevada Gaming Commission standards for slot and video machines.

    • rip says:

      Maybe Estonia could help this poor floundering elephant – the US.
      They’ve been using open-source voting software with full auditable results.
      Of course they have the evil bear over their shoulder and understand how dangerous meddling can be.
      But the will of the Putin lackeys in congress is not going to be swayed by any sense of fairness.

      • Savage Librarian says:

        Several years ago I read that the country is fondly called
        E-stonia because of how esteemed their electronic and digital skills are. I think it was also them that removed a statue that Putin did not want them to remove. Consequently, their electric grid was taken off line by the Ruskies for several days, in retaliation.

  7. harpie says:

    Remember back in 2016 when Richard Burr, the Chairman of this Committee, was a Trump campaign adviser and then he formally joined the campaign on 10/7/16?

  8. Jas says:

    For at least the last 10 years, Republicans have been using the old Russian technique of “always blame your opponent first, in case you get caught “. Preemptive cover if you will.

    • klynn says:

      Yep. Time to dig more on why McTurtle went “political” when he was presented with the IC founded threat on our electoral system. It’s like he knew…

  9. Dopey-o says:

    Any election security should include virtualization of servers, and regularly timed non-erasible backups thru out the day’s election activity.
    Virtua!izing the servers would allow quick restoration of any questionable servers to a certifiable ‘clean’ zero-day state. 60 or 120-minute backups of polling data could reveal patterns of tampering to favor one candidate or another.
    Neither of these are new or untested technology.

  10. mospeck says:

    I sure will miss Spy vs. Spy from Mad magazine. But the spying game has changed. Now it’s happening right out in plain sight. Trump installing Ratcliffe replacing Coats as DNI is on Putin’s to-do list. The thought occurs that Trump has likely even been directed to do it. One of the KGB’s principal aims has always been to undercut, damage, harm US intel. This appointment, and the mid-August “acting” one (in order to avoid a next-in-chain-of-command replacement) are at present being discussed as wrongly politicizing US intel and putting an unqualified person in charge. Agree, but disagree. The Ratcliffe DNI appointment 15 mos. prior to the US general elections is a direct threat to US national security. McConnell, Trump and Barr are doing their best to interfere with the enactment of 2020 election security measures. Spooks and NatSecs have to fight tooth-and-nail against this appointment and also against the “acting” one that is less than 3 weeks away.

Comments are closed.