GAO Analysis Highlights Lab Samples Excluded in Sloppy FBI Anthrax Investigation

As the last Friday before Christmas, late yesterday afternoon was the most obvious Friday news dump hour of the year, and the government didn’t disappoint. The Government Accountability Office released the results of a twenty-three month long study of the genetic analysis that was used to tie the material found in the anthrax attacks of 2001 to the laboratory of Bruce Ivins, whom the FBI concluded (pdf) was solely responsible for the attacks. The FBI’s conclusion is highly suspect for many reasons. On the science side, it is very unlikely that Ivins could have produced all of the attack material on his own and the detailed chemistry of the attack spores suggests that highly sophisticated materials and techniques unavailable to Ivins likely were used to prepare the attack material. Regarding that second point, note that even William Broad refers indirectly to the chemistry concerns in his New York Times article on the GAO report:

To the regret of independent scientists, the report made no mention of an issue beyond genetics: whether the spores displayed signs of advanced manufacturing. They have pointed to distinctive chemicals found in the dried anthrax spores that they say contradict F.B.I. claims that the germs were unsophisticated.

Evidence of special coatings, they say, suggests that Dr. Ivins had help in obtaining his germ weapons or was innocent.

The GAO study was undertaken, in part, because of questions raised by the National Academies study released in 2011 and with special prompting by Representative Rush Holt, from whose district the letters likely were mailed. The GAO study focused on obtaining a better understanding of the validity of the genetic analysis that was carried out and the statistics underlying the conclusions reached.

For a refresher, a helpful illustration from the GAO report shows the underlying biology of the genetic analysis that was carried out in the Amerithrax investigation. Here we see photos of a typical colony of the Ames strain of Bacilus anthracis on an agar plate and four variant colony types that occurred at low frequency when the attack material was spread out on agar so that colonies arose from single cells of the overall population of bacteria that were present in the attack material:
morphs

DNA sequence analysis was employed to identify the changes that led to these variant colony shapes. The FBI then commissioned private laboratories to develop DNA-based tests (relying on polymerase chain reaction, or PCR, methodology) that could be used to screen the large bank of isolates of the Ames strain that the FBI had accumulated through a subpoena submitted to all 20 laboratories known to have isolates of the Ames strain. Developing these assays represented a new frontier in forensic genetics and it did not prove possible to develop tests for all of the mutations identified in the original DNA sequencing. In the end, four tests were developed by the four different contractors.

The Amerithrax report stated that of the 947 samples included in the final analysis, only eight showed all four of the DNA changes the tests were designed to detect. Seven of those samples came from the laboratory where Ivins worked (U.S. Army Medical Research Institute of Infectious Diseases, or USAMRIID) and one came from Batelle Memorial Institute in Columbus, Ohio. The FBI noted that there was a record of material being transferred from USAMRIID to Battelle, accounting for the sample found there.

The GAO analysis finds a number of significant issues with the FBI’s work:

Source of Variant Types

First, the GAO report noted that during the development of the genetic tests, questions arose about the factors underlying the presence of variants and especially whether culture conditions might affect the relative populations of normal and variant types:

Although the specific genetic mutations used as genetic markers to determine a match or exclusion were adequately characterized, the FBI did not conduct studies to understand the methods and environmental conditions that gave rise to the mutations. The FBI convened a team of scientists in 2007 to review the scientific methods. Finding no shortfalls or deficiencies in the basic methodologies they reviewed, they determined that the usefulness of the genetic markers was sufficient. The team also stated that the extent of research and development of the genetic tests at the date of their review was insufficient to determine whether the presence or absence of one or several of the genetic markers was associated with the evidence, was merely characteristic of normal culture practices, or possibly was affected by the sensitivity of detections of the genetic tests. The team recommended additional studies to characterize the genetic markers as a function of growth conditions, including the influence of growth time, growth media, and temperature.

The GAO reports that the FBI’s response to these concerns when they were raised by the NAS panel was hardly encouraging:

In response to questions from the NAS panel about this recommendation, the FBI stated that it considered such studies academic and did not conduct the recommended research.

But that is hardly a just an “academic” question. See this post of mine for a summary of the preparation of Ivins’ RMR-1029 flask, which the FBI treated as essentially a smoking gun. That flask had material from a large number of large scale cultures. Also, the sheer amount of very highly concentrated material in the recovered letters from the attack also suggest very large cultures were carried out to produce the attack material. By comparison, the material submitted by the laboratories in response to the subpoena would be from very small laboratory scale cultures, and so the growth conditions would have been quite different, quite likely affecting the ratios of variant types in the final populations produced.

Sample Submission

Besides the concerns about culture conditions affecting the presence of variants in the samples submitted, the NAS report highlighted a point that had been somewhat obscured previously. It turns out that the scientists responding to the subpoena showed huge variations in how they responded and what they considered to be separate laboratory populations worthy of sample submission:

Our analysis of FBI documents shows that FBI searches at three specific laboratories identified hundreds of additional relevant stocks that laboratories did not submit to the repository in response to the subpoena. Specifically, we found that the FBI collected about 29 percent of the 1,059 repository samples through these searches.

That’s staggering. Nearly a third of the total repository of samples would not have been present had the FBI not searched those three labs. From the Amerithrax report, we do learn that the three that were searched were USAMRIID, Dugway and Batelle. But what about the 17 sites submitting samples that weren’t searched? How many populations were missed in the pool that was tested? The bottom line is that the FBI analyzed a pool of samples that very likely missed a huge portion of what should have been analyzed.

Validation

Very far into the process of developing the DNA tests, the FBI realized they needed to make an effort at validating their analysis. One of the validation attempts put one of the tests into huge question. Table 3 from their report shows this disappointing result:

Table 3

That’s a completely unacceptable result. The test called A1, when run 30 times in a row on material from the “smoking gun” RMR-1029, failed to detect the DNA variation in 13 of those tests. It gave a false negative in 43% of the tests when run on a known positive. And yet the FBI relied on this worthless test as part of the evidence to close the case.

Exclusion of Samples With One Inconclusive Test

If reliance on a worthless test isn’t disturbing enough, the GAO report also dug out a point that was obscure in the NAS report. The FBI stated all along that in carrying out their analysis of the submitted cultures, they chose to eliminate from consideration any culture that gave an inconclusive result on any of the tests. But it turns out that there were some samples that definitely deserved further attention among those that were thrown out:

The NAS report also raised concerns that the decision to remove samples with inconclusive or variant results contributed to the lack of completeness of the repository data. The report stated that a major concern was the restriction of its statistical analyses to the 947 samples that contained no inconclusive or variant results. Notably, the report showed that 4 of the 112 samples that were disregarded for having a single inconclusive or variant result scored positive for the three remaining genetic tests.

Think about that for just a minute. Recall that only 8 of the 947 included samples tested positive for all four changes. And yet there are four more potential samples that might have all four DNA changes that have three positives and one inconclusive among the 112 that had an inconclusive result.

Going back to find that information in the NAS report makes it even worse. It turns out that among the 947 samples included in the final analysis,there were only three that had three positive tests, so the four with three positives and one inconclusive among the excluded 112 is huge. Here is a table with those four samples:
three plus and inconclusive NAS

Where did samples 052-026, 053-010, 054-008 and 054-066 come from? The falsely closed Amerithrax investigation needs to be reopened to follow these sloppily discarded leads.

image_print
15 replies
  1. scribe says:

    Jim, good post. You’ve been one of the only folks following this and have done a great service. But you need to flesh this out far more. It’s been a long time since we’ve been looking at Amerithrax and the finer details have been pushed to the bottom of the pile of many, many details. It would also help to point out, in Layman, why for instance you come to the conclusion:
    .

    That’s a completely unacceptable result. The test called A1, when run 30 times in a row on material from the “smoking gun” RMR-1029, failed to detect the DNA variation in 13 of those tests. It gave a false negative in 43% of the tests when run on a known positive. And yet the FBI relied on this worthless test as part of the evidence to close the case.

    .
    I think I get “why”, but I’ve spent most of the year doing IP cases involving bio-science stuff so my mind is kind of tuned that way right now. The rest of the audience … mmm … maybe not.
    .
    Thanks again for your hard work.

    • Jim White says:

      .
      Thanks. It’s always a very difficult trade-off for me on these posts to find the proper mix of background material and meat of the arguments being made. Too much background makes it sound like a science lesson while running the risk of losing readers before they get to the good stuff. As you point out, too little leaves folks without a basis for evaluating the argument.
      .
      I should do a better job of pointing out that one reason for including the links to the original reports is that they each have pretty good introductory sections that try to present the science that is under discussion. I’d suggest working backwards in time, reading the intro of the GAO report and then the one from NAS and then finally the Amerithrax report.
      .
      The other thing that I do is to try to hang out to provide responses to individual questions like yours. In this case, what is going on is that after the companies were mostly finished developing the methods for their tests, consultants convinced the FBI that they needed to evaluate these tests more fully to see how reliable they were.
      .
      One useful method for evaluating a test that is used to detect something is to run it multiple times on samples that are known to be either positive or negative. In this case, the smoking gun flask was used as the known positive, since it was already known by this time that the population of spores in the flask had some with each of the four DNA changes.
      .
      And yet when the A1 test was run 30 times in a row on material from the flask, instead of all 30 replications of the test coming back positive for the change, only 17 did. The other 13 came up as negative or inconclusive. What good is a test that only works half the time on material of known content?
      .
      Despite that test only being correct 57% of the time in finding the DNA change when it is known to be present, the FBI chose to include this test in its final analysis of the samples under investigation.

  2. Meryl Nass says:

    This discussion is great, but there is so much more that implodes the FBI’s scientific case. Why these morphs? Why spores from not all letters were studied?

    In fact, both now and after the NAS study was published, FBI sang a new tune: the science did not clinch the case, instead look at the totality of our evidence.

    But FBI admits elsewhere that this totality of evidence is completely circumstantial: read, there is absolutely no hard evidence linking Ivins nor flask RMR-1029 nor even USAMRIID to the letters anthrax.

    The bacterial contaminant (B. subtilis) in the first set of letters could not be found at USAMRIID, nor could the materials thought to have contributed to tin and silicon in the spore coat.

    Start again and ask Who Benefited from the anthrax letters?

    Then read Graeme MacQueen’s new book: The 2001 Anthrax Deception. http://www.claritypress.com/MacQueen.html

    Meryl Nass, MD

    • emptywheel says:

      Those are both issues Jim has been at the forefront of examining. This is a post on the GAO report, not a comprehensive post on the anthrax case. Between Jim and I we’ve hit on both the other parts of the scientific holes and on the many ways in which FBI advanced a lone wolf claim based on flimsy and changing circumstantial case to avoid looking for the conspirators.

  3. Don Bacon says:

    The FBI is primarily interested in convictions not justice.
    .
    Gerry Spence, the famous trial lawyer:
    “I found that the minions of the law–the special agents of the FBI–to be men who proved themselves not only fully capable, but also utterly willing to manufacture evidence, to conceal crucial evidence and even to change the rules that governed life and death if, in the prosecution of the accused, it seemed expedient to do so.”
    .
    –From Freedom to Slavery, p. 27
    http://www.amazon.com/Freedom-Slavery-Rebirth-Tyranny-America/dp/0312143427

  4. jaquesDeMolay says:

    it always seemed clear tome that those that were responsible for 9/11 were responsible for the anthrax attack. Just look at who and where the targets were. This flawed cover-up report only gives more weight to that line of thinking. And I am NOT suggestion that Al Qaida had anything to do with the anthrax…

  5. lefty665 says:

    Jim, adding my thanks for your follow up. The FBI’s use of pseudo/junk science to support their preferred conclusion has a long, very long, and sordid history. It needs to be debunked at every turn. Thank you.
    .
    In addition to debunked “science”, the case against Ivins will not stand simple logical scrutiny. The anthrax in flask RMR-1029 was not a secret. It was the USG/DoD certified “gold standard” used to test anthrax vaccines. Its value in validating potential anthrax vaccines was its potency and diverse lineage. It was widely distributed. It is not a surprise that it would be used as the basis for an attack by anyone who knew anything about anthrax.
    .
    Ivins was working for DoD to help protect us from biological warfare. He received the Exceptional Civilian Service Award, the Pentagon’s highest civilian medal, for developing the anthrax in flask RMR-1029. It was not a secret hidden in his lab.
    .
    It seems likely that Ivin’s fatal error was contempt of Agent. The FBI has long treated that as the most unforgivable sin. Ivins was an early part of the search for the source of the anthrax used in the letters. He had a front row seat to the FBI’s bumbling.
    .
    When he was asked to submit a sample, he gave them anthrax from flask RMR-1029. However, he did not provide it in the FBI mandated container. It was rejected, and the FBI demanded a sample in their “official” container. He sent them another sample, in their mandated container, but it was genetically clean, implausibly pure. It seems likely his fatal spiral began there.
    .
    Inferentially it goes like this:
    .
    It is not likely Ivins put the anthrax in a zip lock baggie and threw it over the transom. He had shipped many anthrax samples, both domestically and globally. It would not be surprising to find that he used a familiar and reliable container that had a history of transporting anthrax safely. He was telling the FBI, here’s another, and perhaps better, way to do it.
    .
    When they rejected his award winning anthrax, and exactly the sample they’d asked for, because of the packaging, his response likely was something like: “If you bozos care more about the bottle than what’s in it, I’ll give you the bottle; and, just to show you who’s competent here, it will contain genetically pure anthrax that contains none of the markers you’re looking for. Hope you like the bottle (morons)”.
    .
    Eventually the FBI went back and tested Ivin’s original submission. Viola! It was not the same as the anthrax in the “approved” sample, it was award winning, laboratory standard anthrax, from flask RMR-1029. The FBI’s response: He deceived us! He must be the villain! Even worse, he was jerking our chain and laughing at us!
    .
    That is a capital offense, go gettim’ Efrem. It was all auguring in from there, and dead men tell no tales.
    .
    Keep after the bastards, they’ve earned it.

  6. x174 says:

    funny how the ghost of 911 keeps coming back, now over a dozen years after the fact, with cheney’s depraved defense of government torture, the excessive militarizarion of the police, the metastasis of Islamic jihad in the guise of ISIS, the torture report, and now this GAO report on the anthrax whodunit, which the federal government showed no real interest in except to find some scapegoat to pin it on. thanks for bringing this often forgotten history back to our attention.

Comments are closed.