It’s More than Just WHETHER the E-Mails Are On the Back-Ups

A number of you sent me the AP article reporting that the White House will have to ‘fess up to whether or not the millions of missing emails are on the back-up tapes.

A federal magistrate ordered the White House on Tuesday to reveal whether copies of possibly millions of missing e-mails are stored on computer backup tapes.

[snip]

Facciola gave the White House five business days to report whether computer backup tapes contain e-mails written between 2003 and 2005.

But the actual order is more interesting than that. Here’s what Facciola ordered:

With that understanding, the court will order the defendants to provide answers to the following questions:

1. Are the back-ups catalogued, labeled or otherwise identified to indicate the period of time they cover?

2. Are the back-ups catalogued, labeled or otherwise identified to indicate the data contained therein?

3. Do the back-ups contain emails written and received between 2003-2005?

4. Do the back-ups contain the emails said to be missing that are the subject of this lawsuit?

See, I’m guessing the answer to the more general question–whether the missing emails are on the backup tapes–will be "no." But consider what it would mean if the four questions are answered as follows:

1. Yes, the back-ups are labeled to indicate the period of time they cover.

2. Yes, the back-ups are labeled to identify the data contained there-in.

3. Yes, the back-ups contain e-mail written between 2003 and 2005.

4. No, the back-ups do not contain the emails that are the subject of this lawsuit.

I’m really not sure of number 2 [see the update below for smarter speculation]–or, for that matter, any of my suggested answers. But I think it quite likely the White House will respond (or not respond) in the next 5 days to say that, yes, they know what are on the tapes, but no, most of the missing emails are not on there.

I say that for two reasons. First, review this speculative piece I wrote about when Fitzgerald got particular emails (you know, incriminating ones from Rove to Hadley) during his Plame investigation. I speculated then that Fitzgerald was suspicious about the dearth of emails at least as early as March 2004 (he asked Libby about it), didn’t get the Rove-Hadley email until October 2004 (when Rove explained why he forgot but then remembered talking to Cooper), but didn’t start pursuing the missing emails aggressively until October 2005 (which is precisely when the Office of Administration "discovered" there were a bunch of emails missing). Then, in January 20006, Fitzgerald told Libby’s lawyers that,

In an abundance of caution, we advise you that we have learned that not all email of the Office of Vice President and Office of President for certain time periods in 2003 was preserved through the normal archiving process on the White House computer system.

But he didn’t have the emails yet, not until February 6. So in spite of the fact that (via whatever means) Office of Administration "discovered" in October 2005 that they hadn’t been archiving email properly, they hadn’t gotten Fitzgerald the missing email until January to February 2006, three months later.

So they certainly weren’t able to waltz down to the basement and find the backup tape to reconstruct Rove’s (and Libby’s) missing emails–at least not very easily.

But then there’s this bit, from Gold Bars Luskin (the CNN link on which this was based is dead, but here’s a similar Luskin statement).

The prosecutor probing the Valerie Plame spy case saw and copied all of Rove’s e-mails from his various accounts after searching Rove’s laptop, his home computer, and the handheld computer devices he used for both the White House and Republican National Committee, Luskin said.

The prosecutor, Patrick Fitzgerald, subpoenaed the e-mails from the White House, the RNC and Bush’s re-election campaign, he added.

[snip]

Rove voluntarily allowed investigators in the Plame case to review his laptop and copy the entire hard drive, from which investigators could have recovered even deleted e-mails, Luskin said.

As the investigation was winding down, Luskin said, prosecutors came to his office and reviewed all the documents — including e-mails — he had collected to be sure both sides a complete set.

Now what’s unclear is whether Fitzgerald found any additional emails doing all those hard drive scans, or whether the Office of Administration was able to reconstruct them all themselves (though Jeffress said that Office of Administration is the entity which discovered the OVP emails, at least–does that mean they used a backup tape??). But it seems clear that it was no easy task in October 2005 to just go find emails missing from Rove’s and OVP’s document production.

Which suggests that 1) Office of Administration knows what they’ve got, and 2) at least in 2005, the missing emails weren’t immediately accessible.

Again, the stuff related to Fitzgerald’s investigation is all speculative. But it might suggest that OA is going to have to come back, just in time for the hearing on the destroyed torture tape on January 16, and explain that they do have backup tapes, but that the missing emails are remarkably missing from the backup tapes, too.

In any case, we should know a good deal more in just five days, unless BushCo tells yet another Federal Judge to go fuck himself.

Update: MadDog, who knows a thing or two about computers, says the backup tapes would most likely not be labeled (that is 1 and 2 would be "no").

Based upon my techie experiences, no, the backups are not “catalogued, labeled or otherwise identified to indicate the data contained therein.”

Backups are typically only identified by the date and the system backed up. Content would be unknown other than something as generic as “WH system emails” or “OVP My Document folders”.

The only way that content would be identified would be if someone personally examined each backed-up record or constructed a software program to scan for certain keywords (kinda like how one would imagine the NSA would scan for stuff on all the databases that were warrantlessly eavesdropped upon).

Which brings one to the real hot fact: If someone in the WH is claiming that specific stuff is missing (i.e. Rove’s Abramoff involvement, various parties including Rove’s involvement in Valerie Plame Wilson’s betrayal, etc.), then be sure that they have done that scanning to arrive at that position.

You can’t have that kind of specificity without having done the dirty work to find out just what is on the backups.

And here’s William Ockham, who also knows a thing or two about computers:

The answers to 1 and 2 should be straightforward. The answer to 3 will be interesting. I would expect by this time the answer to 4 would be some of them.

Btw, the WH has spent some money this year on consultants who should have been able to help them.

Thanks to both MD and WO.

image_print
105 replies
  1. FrankProbst says:

    Suffice it to say, it’s going to look really bad if only a select number of e-mails were “not properly archived”. And no judge is going to be amused by the answer, “We have all of the documents, except for the ones you’re asking for.”

    • emptywheel says:

      Yeah, I guess that’s my point all boiled down into one coherent sentence. I kind of imagine someone saying,

      Well we have all the emails except the ones tied to the Abramoff scandal, the Plame outing, the DeLay Texas redistricting, the NH phone jamming follow-up, the AIPAC leaks, the Chalabi discussions, the energy task force, unnamed scandal A, unnamed scandal B, unnamed scandal C, unnamed scandal D, and a few more random acts of corruption. But aside from those, yeah, we’ve got all your emails.

      • FrankProbst says:

        Well, we know that answer to #4 is “No”, because Fitz wouldn’t have left a stone unturned to find those e-mails. If Fitz and the FBI couldn’t do it, I doubt a federal magistrate is going to have any luck. I’m still not sure your take on #1-3 is accurate. This administration tends to rely on looking unbelievably incompetent in order to avoid looking guilty. But I’m not entirely sure they can pull that off here. As Fitz pointed out, there was some sort of “normal archiving process” in place. Somebody had to actively screw it up. Which means someone is going to have to admit to active “mistakes were made” incompetence, as opposed to passive “no one could have anticipated” incompetence.

        • emptywheel says:

          Well, one scenario in which 4 is “yes” is that they recognized the archives didn’t have the emails, and had to reconstruct them, using backup tapes, which is where they got those emails.

          Though if that were true, wouldn’t you just rebuild the archive?

          I don’t know. I’m going to find William Ockham who is good at these things.

          • WilliamOckham says:

            I’m swamped at work at the moment. I’ll have some input on this stuff later this evening. The answers to 1 and 2 should be straightforward. The answer to 3 will be interesting. I would expect by this time the answer to 4 would be some of them.

            Btw, the WH has spent some money this year on consultants who should have been able to help them.

        • bobschacht says:

          “Well, we know that answer to #4 is “No”, because Fitz wouldn’t have left a stone unturned to find those e-mails.”

          What if the answer has changed since Fitz asked? That is, when Fitz asked, the available info on the backups was so hopelessly general, or even incorrect, that it was impossible to be responsive, but since then, they’ve done more indexing work?

          In other words, when is “Not that we know of” an acceptable response?

          Bob in HI

      • klynn says:

        EW,WO,MD,

        Thanks for the great post. Oh how I would pay big bucks to sit in a judges chambers to hear those words spoken out loud!

        I wonder how the government does their “off-sight” back-up? Is it privately contracted? We could probably find the RFP and the bids…

        O/T I did not think I would be reading the threads for a while but my day just got more interesting. My daughter is under my constant observation here at home until further answers on her health issues. My 15 year old son came home from an indoor soccer game at school with a concussion and a broken nose! WE just returned from the hospital. Poor kid. SO, I will be awake all night checking on them. I hope the threads keep me awake. bmaz,and all, thanks for your kind words this AM.

        I needed this thread to lighten the day!

        • MadDog says:

          I wonder how the government does their “off-sight” back-up? Is it privately contracted? We could probably find the RFP and the bids…

          Here’s what I’ve been able to find out in a few quick googles:

          Office of Administration

          The Office of Administration was established by Executive Order on December 12, 1977. The organization’s mission is to provide administrative services to all entities of the Executive Office of the President (EOP), including direct support services to the President of the United States. The services include financial management and information technology support, human resources management, library and research assistance, facilities management, procurement, printing and graphics support, security, and mail and messenger operations.

          the Office of Administration’s Major Information and Record Locator Systems…

          …AUTOMATED RECORDS MANAGEMENT SYSTEM (ARMS)
          Serves as an internal storage for capturing electronic mail messages created and received by the offices and councils within the Executive Office of the President….

          I’d have to bet dollars to donuts that backup/archive media is transported off-site. And probably to a “secure” storage site since national security info is contained on the media.

            • MadDog says:

              I’d expect there are tons of secure sites located all over the place nearby DC. Considering all the DC DoD, Intell and DoJ/FBI population, how could there not be.

          • jayackroyd says:

            Picking this one at random to reply to, one should be correctly gathering the impression that saying the backup tapes were destroyed is pretty unlikely if the WH was running even a reasonably standard IT operation. First, there’s the rotating set of daily backup tapes in the jukebox. Then there’s the weekly (or monthly) off-site. Then there are the retired dailies that are stored away somewhere.

            Moreover, you can’t know you’ve lost backup material because a lot of people retain their email indefinitely. IME, the more senior they are, the more likely they are to do this, and not be pestered by the IT department to delete mail to save space.

            So even if you do find and destroy the dailies for a particular time period, retrieve the off-sites for the same time period (and keep in mind that there will be records of that; the offsite vendor will insist on those), you *still* can’t be sure you’ve destroyed the mail.

          • klynn says:

            Thanks so much! Mr. Klynn is cutting a business trip short to get back for the kids. he had quite the day. The flooding in Indiana caused a car to hydroplane going into the toll booth and he was rear ended in his rental car! I should have bought a lottery ticket!

      • rhfactor says:

        Marcy, back when the RNC emails were first requested by Waxman on March 21, 2007, a flurry of diaries were written at DailyKos (in addition to your writings, and TPM’s). Many contained commentary from IT specialists regarding various points of redundancy besides backup tapes. Below are the URLs of a couple of these forensic recovery threads. One question I have is: Whatever happened to the idea of scouring the Blackberry servers given the vast majority of those emails were sent to/from Blackberrys?

        You have been the primary point person for continuity and analysis of these missing emails… In a nutshell, what has happened to information provided in all those threads from the wide array of IT specialists who articulated nodes and various methods of reconstructing email BESIDES server tapes?

        1. Slashdot Techie opinions on White House e-mail scandal
        by CTMET [Subscribe]
        Sat Apr 14, 2007 at 01:05:13 PM PST
        http://www.dailykos.com/story/…..161/323239

        2. Missing White House E-mail Forensics
        by Mash [Subscribe]
        Wed Apr 11, 2007 at 10:29:13 PM PST
        http://www.dailykos.com/story/…..226/322157

        3. I Learned the Hard Way that E-Mails Are Forever: They Never Really “Disappear”
        by Jesselyn Radack [Subscribe]
        Thu Apr 12, 2007 at 02:51:47 AM PST
        http://www.dailykos.com/story/…..223/322184

        4. 95% of Rove’s emails from RNC servers=The Endgame
        by BloggerJohn [Subscribe]
        Sat Mar 24, 2007 at 12:20:46 AM PST
        http://www.dailykos.com/story/…..835/315446

        5. (your own – one of many)
        Waxman Says: Give Us the RNC Emails
        by emptywheel [Subscribe]
        Wed Apr 04, 2007 at 10:08:19 AM PST
        http://www.dailykos.com/story/…..929/319420

        6. “Lost” White House Emails Are NOT Lost (a note from your friendly, neighborhood IT Guy–Updated)
        by jpadgett [Subscribe]
        Wed Apr 11, 2007 at 07:13:42 PM PST
        http://www.dailykos.com/story/…..292/322099

        7. Memo to Sens. Leahy and Specter: Choose Your Forensic Expert Wisely
        by wanderindiana [Subscribe]
        Sat Apr 14, 2007 at 01:41:13 PM PST
        “Making the right choice on forensics is going to make or break this investigation and any subsequent prosecution. Don’t screw this up.”
        http://www.dailykos.com/story/…..170/323246

        8. Deleted Emails May Rise From The Dead
        by Horsefeathers [Subscribe]
        Wed Jun 20, 2007 at 04:13:56 PM PST
        http://www.dailykos.com/story/…../24/348817

        9. Update from Spkr Pelosi: Committee Orders All RNC Emails To Be Preserved
        by Scout Finch [Subscribe]
        Mon Mar 26, 2007 at 09:31:27 AM PST
        http://www.dailykos.com/story/2007/3/26/12271/2545

        —–

        Marcy, I know you have a way of professionally detaching yourself from the emotions that surround various stories — which enables you to connect the dots so well. But in this case, I just want to ask you: Are you not pissed royally by the poor management of this RNC email issue by the De,. oversight committees? To me they have been baffoons going on 9 months. Somehow, I can detach from Rove and Gonzo and many other things — but this to me, from the get go, was the hard evidence we always sought re conspiracy to obstruct justice. Based on all the expert “testimony” from wide variety of IT people contained in those threads above, it seems inconceivable to me that the g.d. oversight committees have not done their own forensics recovery by now. It burns me up.

        • klynn says:

          Most definitely. I shared a story about my work in the Presidential election in Ohio in the FDL threads (a long time ago) that pointed to the use of “phone gathered information” (surveillance) in terms of the Ohio election. My guess is there are emails out there that spread the information gathered on Ohio strategy by Dems and independent groups..

  2. PJEvans says:

    Did they label the backups before or after they tried finding the e-mails? Because I can see the labelling being done (possibly not well) as they did the searches, or the labels ‘accidentally’ getting separated from the tapes somewhere along the line.

    • emptywheel says:

      Obviously, I have no clue. But I guess the point being, OA seems to have had to go through and find a bunch of emails (rather 250 pages, at least) in 2005-2006. So if the emails DO exist on the back-up tapes, then they may have simply taken 3 months to go through everything and label them.

      • PetePierce says:

        If the Court, specifically Magistrate Judge John Facciola in this case were serious about finding out the truth, the Court could have ordered the FBI under seal to raid and seize and examine those hard drives for a specified period of time, and they could have done it in a timely fashion.

        But the Court didn’t. Because the court is not serious about finding out the truth.

        Who thinks anything susbstantive can come from this order? You don’t order this White House to tell the truth on planet earth. OA or Office of Administration is not about to give up anything that hurts Bush Co. It has become abundantly apparent on every issue that this White House is going to cover up. And pretty consistently this DOJ is going to run interference for them when they cover up.

  3. Bushie says:

    Of the many IT people I’ve known, two traits are common. 1. no sense of humor. 2. Honesty to the extreme. I wonder what would happen if investigators, etc. talked to the people in the trenches about missing backups.

    • MadDog says:

      Of the many IT people I’ve known, two traits are common. 1. no sense of humor. 2. Honesty to the extreme. I wonder what would happen if investigators, etc. talked to the people in the trenches about missing backups.

      You must’ve been hangin’ with the wrong IT folks. As one for 25+ years, we generally have more than our share wrt a sense of humor. Only one of us could ever have come up with Dilbert.

      Of course, only one of us could also have come up with the Simpsons, so I’m not qualifying us IT techies as mature, rational adults. *g*

  4. oldtree says:

    I don’t understand that language either. Is he asking the white house to destroy the backups so that he need not ask for them again? The wording is bizarre at best.

  5. Tross says:

    EW,

    Why do you think the delay in turning over the emails to Fitz was an indication that they couldn’t be found quickly?

    Wouldn’t this administration drag its feet as long as possible under the pretense of “looking for them” and turn them over as late as possible?

  6. MadDog says:

    “I’m really not sure of number 2…”

    Based upon my techie experiences, no, the backups are not “catalogued, labeled or otherwise identified to indicate the data contained therein.”

    Backups are typically only identified by the date and the system backed up. Content would be unknown other than something as generic as “WH system emails” or “OVP My Document folders”.

    The only way that content would be identified would be if someone personally examined each backed-up record or constructed a software program to scan for certain keywords (kinda like how one would imagine the NSA would scan for stuff on all the databases that were warrantlessly eavesdropped upon).

    Which brings one to the real hot fact: If someone in the WH is claiming that specific stuff is missing (i.e. Rove’s Abramoff involvement, various parties including Rove’s involvement in Valerie Plame Wilson’s betrayal, etc.), then be sure that they have done that scanning to arrive at that position.

    You can’t have that kind of specificity without having done the dirty work to find out just what is on the backups.

      • MadDog says:

        If they had to go through and find stuff, though, would they catalog it?

        Ahmmm…Yes and No. *g*

        One would catalog what one was searching for and finding, but not catalog what one wasn’t looking for, and therefore not finding. Does that make sense? *g*

        I mean, if you were searching for content wrt to a keyword such as “Plame” or “Abramoff”, you would catalog that.

        What you wouldn’t catalog, if you weren’t searching for it, would be stuff like “Junya OKs torture”.

        If you don’t “look” for something, rest assured, you will not be finding it. *g*

          • MadDog says:

            But would you do something like, “OVP backup, June to July 2003 [This is where we found Dick’s secret order to out Plame]”?

            Absolutely! This would be a perfect example of “labeling” a backup.

  7. marksb says:

    I’ll back up MadDog. Techies have a well-developed sense of humor, but it’s often so dry and obscure (snark in code comments, the names given to servers, responses given to the CFO in project meetings and the CFO hasn’t a clue what it means or if he was just made fun of, judging from the smirk on half the faces in the meeting…) that it goes over the non-techies head.
    Dead-on on the backups, too. The only way they know the files are missing is if they’ve done a content analysis.

    • bobschacht says:

      I’ll back up MadDog. Techies have a well-developed sense of humor, but it’s often so dry and obscure (snark in code comments, the names given to servers, responses given to the CFO in project meetings and the CFO hasn’t a clue what it means or if he was just made fun of, judging from the smirk on half the faces in the meeting…) that it goes over the non-techies head.
      Dead-on on the backups, too. The only way they know the files are missing is if they’ve done a content analysis.

      I’ll add my “Amen,” as a former programmer who used to write in Fortran IV, Cobol, Dbase III, and (heaven help me) even in machine language. There are two schools of programmers: some who have been brought up to write everything in clear, functional language with simple, explicit explanatory comments (The IT equivalent of Dudley Do-Right), and others who are far more devious, and label everything (including variable names, file names, and everything else) with code words that only they understand. The motivation is not so much a desire to cheat as it is a desire to make oneself indispensable. One can fire the Dudley DoRight’s in a New York minute, secure in the knowledge that they’ve left everything in a Ready-To-Use format, reducing transition time to nil. If you fire the devious programmer, however, you’re faced with the possibility that you may have to start from scratch because it may be too difficult to decipher what the Delphic One has left behind. Transition costs can be enormous.

      Segue over to our present scenario. An IT guy could start out as a Dudley DoRight, and then find oneself in the position of Sybel Edmonds. What to do? One’s first reaction, ordinarily, is CYA– perhaps at first on a minimal scale, since you only sense personal jeopardy on one small item. Then you start to become aware of how “friends” are compromised. So the CYA system starts to expand. Pretty soon the CYA has rendered the records nearly impossible to decipher by anyone else.

      But that’s my projection. marksb, am I hitting anywhere close to home?

      Bob in HI

  8. PJEvans says:

    MadDogs preety much confirms what I was thinking (having spent several years going out with a computer consultant/geek whose home computer was labelled DEC).

    Normally backups are only labelled with the date and, if necessary, the computer’s ID. Anything else – well, the really thorough might print the directory and tape it to the package or file it someplace else so you could actually retrieve the data. Somehow I don’t think the WH did that, or if they did, they won’t be admitting it.

    • MadDog says:

      Normally backups are only labelled with the date and, if necessary, the computer’s ID. Anything else – well, the really thorough might print the directory and tape it to the package or file it someplace else so you could actually retrieve the data. Somehow I don’t think the WH did that, or if they did, they won’t be admitting it.

      Most backups (perhaps even all these days) are done via a specific software program that does backups and only backups. There is some amount of operator control over the “labeling”, but generally it is generic such as “Exchange Server XYZ from 01/01/01 to 01/02/01).

      Content can really only be determined by “Google-like searches” or by reading the individual records.

      “Google-like searches” are what is typically done on backed-up email for legal discovery stuff, so the “art” is making sure your searches are narrow and really specific.

      An example would be searching for instances of references to “waterboarding”. If the writers used instead the hyphenated “water-boarding”, your search would fail.

      If instead you searched for “water” and “board”, you’d get your hits.

      You’d get stuff like “I spilled water on my laptop” or “I was appointed to the board of Mothers Against Drunk Repugs”, but you’d also hit paydirt with stuff like “We sure waterboarded that sucker. He gave up everything on Clinton’s strategy.”

      • emptywheel says:

        I guess it’s all a question of what you mean by content. I think a backup specific to Office of Political Affairs is specific enough to be interesting (not least because most of the people in OPA have had their email mysteriously disappear). But I don’t expect they’ll have it labeled to the granularity of “This is where JimmyJeff GannonGuckert promised to suck Rove’s dick,” at least not on the outside label.

        • MadDog says:

          While I don’t know how the WH IT is setup, some of the emails I’ve seen lead me to believe they are using standard Microsoft Exchange Server for their emails.

          Most large corporate organizations tend to have “centralized” IT versus “distributed” or “department” IT. Economies of scale and all that.

          If the WH IT operates like most corporations, then there is likely not any IT demarcation between various WH departments.

          For example, all email services would be handled in a centralized fashion, so that you’d likely not have an email server dedicated specifically to a single department (like one for the OVP and another for the EOP).

          That said, the size of the population served determines the number of email servers. If you have 20 people being served, that can be handled on 1 email server.

          If you have 1000+ people being served, you’d spread that load across multiple email servers. And you also then likely have multiple daily backups of these servers (for example backup 01/01/01 to 01/02/01 for email server A, the same for email server B, C, etc.).

          And added to this mix may be the requirement that certain folks by dint of their national security work, have a secondary national security email account which is not the same as their normal WH email account. And that secondary national security email account would “likely” be housed in a physically separate (and secured) email server.

          We IT folks can make things as complex as need be, and often even if not needed. *g*

        • MadDog says:

          But I don’t expect they’ll have it labeled to the granularity of “This is where JimmyJeff GannonGuckert promised to suck Rove’s dick,” at least not on the outside label.

          LOL! No, that would likely be only on the inside of the Hallmark greeting card that Deadeye keeps in his walk-in safe.

  9. PetePierce says:

    I don’t expect any surprises here. The Judge can order, people can jump up and down, and if emails have been destroyed or backed ups destroyed, you’re not going to ever know.

    Why didn’t the Court simply order seizure of their hard drives for a specified period of time under seal, and let the FBI surprise them.

    It’s possible. The Court could have done that. But instead the Court is taking the same passive naieve posture.

    They are trusting the White House to answer honestly.

    When there are criminal defendants, you can bet your bippy that raids are done.

    No raids here though are there?

    • JohnForde says:

      And OVP won’t have to admit to the active ’someone screwed up” line. It was spontaneous combustion … in the ceremonial offices …. where they keep the backup tapes.

      • PetePierce says:

        Yes indeed.

        I really appreciate EW, as she often does so well, shining light into the crevices where so much coverup is going on. It always informs me further and and makes me want to get better educated in that area. But everytime I see references to one of these situations, phrases like “Pixie dust” come to mind.

        This is an administration that laughs at the Court system as far as orders for them to do anything. If and when the Supreme Court rules against them, they just opt themselves out (in their view with a signing statemennt or two or brush it off). When they don’t see consequences to their behavior, they are going to do whatever they want.

  10. skdadl says:

    A small factual question about these paragraphs in the AP story:

    Two months ago, the two private groups focusing on issues of government secrecy obtained a court order directing the White House to preserve the backup tapes.

    The Bush administration had offered to have the government file a sworn declaration stating that the White House is safeguarding all backup materials. Instead, Facciola recommended that the judge in the case, Henry Kennedy, issue a court order because “a declaration is not punishable by contempt. In other words, without such an order, destruction of the backup media would be without consequence.” Kennedy did as Facciola suggested.

    I’m trying to figure out the relationship between Judge Facciola and Judge Kennedy. I thought I understood Judge Kennedy’s role; how did Judge Facciola come into the case? (Sorry: ignorant alien here.)

      • bmaz says:

        I have no knowledge about this specific case, but district courts that have burdened dockets (that would be all of them) generally operate as follows: Criminals cases have preference (by law and per Sixth Amendment guarantee); then civil cases. On certain civil cases, especially ones that are amenable to a mediated resolution, the assigned district court judge will farm the matter out for this purpose to a magistrate judge. The case remains assigned to the district judge though and, absent some type of semi-permanent special master appointment, the district judge will still make all substantive determinations and orders.

  11. Hmmm says:

    Kinda getting into the techno-semantic weeds here, but: Different offices use different back-up strategies (some wisely, others less so), and in some strategies there is a distinction between the “back-up” and the “archive”. Typically “back-up” data is stored on a few sets of media (tape, CD/DVD, hard drive, storage at a remote location sent over the internet, what have you) that get written over (old data erased) frequently, usually nightly; for example, rotating through 3 sets of tapes, which might be labelled set A, set B, & set C. This is for getting back up and running quickly when a computer fails, so that only new work done since the last back-up is lost. Whereas “archive” media act as a snapshot of the state of the computer(s) as of a given date; “archives” media are added to some permanent-storage closet somewhere as institutional memory. This is for making sure the organization does not lose critical data because somebody mistakenly erases it without noticing (the “back-ups” would miss that), and secondarily in some offices as a way to prove the state of things as of a particular date. “Archive” media is always labelled, that I’ve seen.

    Point being, “back-up” media may well not be meaningfully labelled, and may well not go back very far in time, whereas “archive” media might be what’s needed here. So, the order wording by focusing in on the term “back-up” might leave too much wiggle room.

    • PJEvans says:

      Even that doesn’t always preserve stuff.

      The company I work at does just about everything it can electronically these days, and we still find stuff that dropped out of the system somehow. The last one we couldn’t find when it vanished, even – they went back at least three years looking for the particular file that disappeared, and couldn’t find the previous version (but we had a file name to start with).

      • Helen says:

        Yes – but I’m sure the losses were data created within a short timeframe. What the losses weren’t were data over a longer period of time, and more importantly, only about one specific subject.

  12. CTuttle says:

    Aren’t the bulk of the RNC servers located in Chattanooga, Tenn… I seem to recall something about that during the ‘04 election… Ohio’s votes inparticular…

  13. SaltinWound says:

    I don’t know that Fitz wouldn’t have left a stone unturned to find emails. I also don’t know why he waited until October, 2005 to aggressively pursue missing emails.

  14. billdurbin says:

    No one has mentioned the posting on the National Security Archives web site. They are one of the parties in the consolidated case with CREW.

  15. freepatriot says:

    first off, this is a BACKUP TAPE

    it is designed TO BE USED if the system fails or destroys the data

    so this isn’t some random data set

    this is designed TO BE RELOADED INTO A COMPUTER

    so just reload the fucking tape, hit “Open All Documents” and find out what each bit of data is

    it might actually BE rocket science, so why not HIRE SOME ROCKET SCIENTISTS

    i don’t give a shit what george bush says is one these tapes

    I want the fucking tapes

    george can offer his views, under oath, and with the full penalty of perjury in effect

    then we can get some techies, and actually FIND OUT WHAT’S ON THESE FUCKING TAPES

    then, if there are SELECTIVE OMMISSIONS, we can deal with that once we KNOW there are selective ommissions

    absence of a thing somethimes tells you all you need to know

  16. WilliamOckham says:

    I’m very disturbed by the definition of back-up that the judge uses:

    [media] that were created with the intention of preserving
    data in the event of its inadvertent destruction, and that are being preserved in accordance with Judge Kennedy’s order.

    There’s a lot of room for the WH to claim that they don’t have any of that while still having the missing emails. As just one example, I would expect them to have disaster recovery hardware that exists not to preserve data, but to have systems ready in case of disaster. Often those systems have some data on them. Don’t forget a lot of these emails got cc’ed or forwarded to other parts of the administration or outsiders. The WH could have gathered that kind of stuff, but it wouldn’t fall under the judge’s definition of “back-ups”.

    • emptywheel says:

      Though the logic is that, if they don’t have the emails on the backup tapes, then we can all begin fighting about the other things like disaster recovery hardware.

  17. Neil says:

    Kennedy to Mukasey Ltr 1/8/8
    RE: criminal investigation into the destruction of the CIA interrogation tapes

    Regarding the scope of the investigation, some have interpreted your announcement as limiting the investigation to crimes committed in destroying the tapes. However, the investigation must also include an effort to recover and examine what was on the destroyed tapes. Surely, the content of the tapes may prove highly relevant in determining why the tapes were destroyed. If the conduct depicted on the tapes involved the commission of crimes, it, too, must be pursued. America’s laws against torture are every bit as important as our laws against obstruction of justice, and they too deserve the vigorous enforcement that you pledged in your confirmation hearings. As in any truly independent investigation, the investigators must be allowed to follow the evidence wherever it leads and to bring any wrongdoers to justice. Please let me know if this understanding of the scope of the investigation is correct.

      • Neil says:

        …Still atoning for youthful and middle aged indiscretions no doubt. I believe he’s one of the most prolific legislators, as in a senator who authors legislation. Of course, we can credit him for Medicare and “borking” as a verb.

        He held a panel discussion on No Child Left Behind today with education experts. He wants to improve on it. Bush is on record that he’ll veto the bill if there are changes to accountability for test results.

  18. nonster says:

    Since there are policies about preserving these records for posterity I wouldn’t assume that they use only off the shelf commercial email systems. I’d expect a system that archives emails in a way that is oriented towarded cataloging the entire administrations’ email output for review in the future, wothout needing a specific software product to interpret it. Otherwise there will be a huge amount of IT support necessary to support future research of these materials. Or if they’re lucky they’ll find an ancient copy of Microsoft Exchange Server 2000 and the hardware to run it on.

  19. kspena says:

    I have a memory of the ‘old lion’ to share. Forth-eight years ago on a very cold day, I was at a ski-jumping competition near Madison, WI. During a break between the first and second jumps the announcer said something like, “We have a special treat today.” He announced that the youngest brother of the presidential candidate, John Kennedy, was going to jump. He had NEVER jumped before. The young man was doing it on the wager that if he made it, we all had to vote for Jack. Teddy, in his heavy brown coat and floppy pants made an extrodinary jump. Not great form, but he made it. The Kennedy name became a household word in Madison that day….

    • bobschacht says:

      I have a memory of the ‘old lion’ to share. Forth-eight years ago on a very cold day, I was at a ski-jumping competition near Madison, WI. During a break between the first and second jumps the announcer said something like, “We have a special treat today.” He announced that the youngest brother of the presidential candidate, John Kennedy, was going to jump. He had NEVER jumped before. The young man was doing it on the wager that if he made it, we all had to vote for Jack. Teddy, in his heavy brown coat and floppy pants made an extrodinary jump. Not great form, but he made it. The Kennedy name became a household word in Madison that day….

      I grew up in Madison, and must have been there at the time, but I don’t remember anything about this. Heck, I don’t even remember that we had any hills high enough to make a ski jump. I would love it if you could come up with a date and a link to a Madison newspaper story about this event.

      BTW, I’m not saying it didn’t happen. I was a teenager at the time, and lord knows my mind was probably on other things…

      Bob in HI

      • phred says:

        Bob — I think the place in question might have been Devil’s Head. It was the only downhill bump around. We had a class trip there when I was in the 8th grade. Suffice it to say a lot of us flatlanders never made it off the bunny slope that night. And one of us, never really even got the hang of the tow rope ; )

  20. Neil says:

    Live primary results at thr Concord Monitor
    updated every 5 mins.

    Clinton , Hillary Dem 72,227 39%
    Obama , Barack Dem 66,010 36%
    Edwards , John Dem 30,877 17%
    Richardson , Bill Dem 8,565 5%
    Kucinich , Dennis Dem 2,595 1%
    (69% in)

    McCain , John GOP 54,757 37%
    Romney , Mitt GOP 47,231 32%
    Huckabee , Mike GOP 16,824 11%
    Giuliani , Rudy GOP 12,783 9%
    Paul , Ron GOP 11,557 8%
    (64% in)

    Good turnout again by Democratic voters relative to Republican voters.

    • emptywheel says:

      It has been called for Hillary, with those same percentages between her and Obama.

      Things get very very interesting from here. I just wish MI wasn’t such an everloving clusterfuck.

      • bmaz says:

        Do you really sense an effort by the Edwards and Obama camps to pump up the undecideds as a category against Clinton or is that more hype?

      • Neil says:

        Political coverage in the press, like football fans, take one game or one primary contest or two as their data points and extrapolate to the final outcome.

        We have a split right? Let’s hear what democrats in other parts of the country think.

        I’d like the top three candidates to make it to the convention. I know its not going to happen but I think it’d be good for the party and keep the Republican op-research folks on their toes.

        What is it about the MI primary that bumming you out?

        • emptywheel says:

          1) The DNC has said it will strip MI of all of its delegates. And only Clinton and Kooch are still on the ballot. That should say Clinton will win something that has promised to be meaningless.

          2) But everyone knows the DNC will recant once we have a nominee, and give us our delegates back, with the assumption they will support the nominee. So that says it does have meaning, kind of.

          3) That gives an incentive to those who support Obama or Edwards to vote. but they have to vote “uncommitted” to have their vote count–that is, not count for Hillary.

          4) Meanwhile, the GOP has stripped the GOP of just HALF of its delegates. That means MI, with its 30 remaining GOP delegates, is a very very meaningful contest, and particularly interesting for a contest between McCain (who won here in 2000) and Romney (whose Daddy was a very popular governor here).

          5) But McCain won in 2000 with the help of TONS of cross-over voters (it’s very easy to vote in the otehr party’s primary). I expect that a lot of people, seeing the Dem vote as virtually meaningless, will cross over.

          So while it could be a really remarkable contest, on both sides, I think the McCain Romeny contest will be skewed by Dems, which will in turn skew Dems, as will the uncommitted requirement. And in Hillary’s case, it’ll offer her an opportunity to claim she solidified her lead coming off NH, which might or might not be true, given the condictions.

          • phred says:

            EW, I’ve been wondering about the MI ballot — how come none of the rest of the Dems are on it? If NH can put O. Savior and Vermin Supreme (apparently for $1000, you too can put a silly name on the NH ballot), how come legitimate names are kept off in MI? What’s the process for you folks?

            Oh, and more on topic, from my days as a defense contractor, I know we had an off-site warehouse for long-term data storage. It seems to me that if 10 million emails went missing, it was because someone made a point to make them to disappear, whether by using alternative non-government accounts and servers (thanks RNC!), disabling the back-up/archiving software for government accounts and servers, or by “losing” the tapes en route to storage. And I also concur with others here that there is a distinction between short-term backups and long-term archives. I can’t imagine the WH successfully parsing that distinction before a judge though.

            • bmaz says:

              It seems to me that if 10 million emails went missing, it was because someone made a point to make them to disappear,

              Thank you Phred. And no shit! It has always blown my mind how nonchalant the sheer huge number of missing emails has been blithely treated by the media, the congress and everybody else. This was not the government as a whole, it was just the White House. I cannot remember what the relevant time period was, and I do not know how many individual people/accounts this figure covered. But the “White House” is not that big; lets use 200 people/accounts. Lets make the timeframe big to give the benefit of the doubt; say two years and take a middle figure of what has been bandied about, at 5 million emails. That works out roughly to 35 emails a day, every single freaking day including weekends and holidays, for every stinking one of the 200 people, that were deleted. This is insane.

            • emptywheel says:

              When the DNC took away our delegates, the candidates made a deal not to campaign (so MI wouldn’t get any cash), and most pulled off the ballot to be “good Democrats.” Dodd, Hillary, Kooch, and (I think) Gravel remained on the ballot, which is why and how Hillary is still there.

              Incidentally, David Bonior (former MI Congressman and governor candidate with a still active machine in SE MI) is one of Edwards’ top advisors. And, at least in the early fall, more people from Washtenaw were on Obama’s lists than any other county (or something like that). So both–particularly Edwards, with the labor vote here–turned down a lot by not running in MI.

              • phred says:

                Nothing like cutting off your nose to spite your face, eh? So what do you think, will there be a push to get folks to vote “uncommitted” as a proxy defeat of Hillary? Or do you figure people will opt for one of the names that remains?

                I also find this interesting in light of your observation after IA that Hillary does care about the party. Apparently not enough to be a “good Dem” and take her name off the MI ballot ; )

          • Neil says:

            Thank you. You have to wonder if the DNC’s decision about MI’s primary doesn’t hurt the DNC more than the Michigan.

            Take a look at the price of “Clinton wins the Democratic nomination” on Intrade in Krugkman’s blog post
            Nobody knows anything“.

            • jdmckay says:

              For example: let’s say the organization only requires a FULL backup once a week, after hours so as not to bog down system and network performance. It gets run each Friday night. By the end of the month, I have 4 full backups; the only difference between this week’s and last week’s back up is the emails sent that week. If the system runs differentials every day, they only back up each night after hours the new content created that day — but you can see where by Friday there should be 5+ copies of the emails sent on Monday.

              Two pertinent points…

              * Hardware: the backups (archives) would have redundant storage… at least 3 distinct media devices, 5 is common now (disks are so inexpensive), 7 not unusual. The data is written simeoultaneously to each storage device (whether disk, tape, etc.). Lot’s of hardware companies market canned storage hardware (eg: RAID/STORAGE MEDIA/MNGMENT SOFWARE). Custom configuring this stuff (as opposed to “canned” storage)is very well established, not terribly sophisticated procedure. For many years, I provided such setups (my biz is primarily custom software) for customers w/specific needs. Point being, loss of data from storage failure is not something that happens these days. Most larger companies will even have redundant media @ different locations to protect against fire damage and such.

              * Software: all the commonly used mail servers have various means by which administrators can monitor reliability of storage. Most of the better ones (which I’d expect to be used in something like WH) notifies admins when something goes wrong, whether hardware, network problems… whatever. Or in other words, they know when something’s not working properly (eg: backup) almost immediately. Most of ‘em have notification to the admin build into the network: eg. anything that goes wrong is immediately put in email and dropped into admin’s mail box. Beyond that, when archiving is known to be needed for accessability… eg: searches, it is also very simple, well understood practice to store/restore individual emails in a well organized database (db): this means search facilities of db can be put to use. I can’t imagine WH wouldn’t be implemented in this fashion. All the “server” db(s) (Oracle/DB2/MSQL/POSTGRESS etc.) manage their own “sections” (called pages) on the storage media. This means that stored archives are accessable irrespective of “group”: eg. there is no “this week’s backup” as opposed to “last week’s backup”… it’s all in same datastore (db), each record as easily accessable as any other. The organization of data across “pages” managed by the db software itself as opposed to being categorized by this/last week (month): eg. the db software creates & mangaes it’s own “groups” of data on the storage media. For purposes of retrieval, these “pages” are invisible… as I said, entire db contents are equally acessable. As with the hardware, any of these db(s) would immediately notify admin if anything went wrong: disk corruption, RAID problems, bad writes (media hardware failure)… all this monitoring is built into these databases. In a redundant storage array w/5 media storage devices, the failure of only one device would immediately trigger an admin notification, all the while the other 4 devices would function normally. In most of these “arrays”, a failing device can simply be lifted out and new one put in (”hot swap”) and backup hardware/software will automatically build records on newly installed media.

      • MadDog says:

        Things get very very interesting from here. I just wish MI wasn’t such an everloving clusterfuck.

        Think of it instead as a everloving evolving clusterfuck. I truly believe that come convention time, things are gonna be forgiven.

  21. MrWhy says:

    IT types distinguish between archives and backups

    backups are copies made for disaster recovery purposes, and due to data volumes, backup retention windows vary widely. Think 1 year as normal.
    archives are files which are known to have long term value so efforts are made to ensure long term availability

    Policy will often dictate what should be archived.
    This is one reason why Rove and others had multiple email addresses, so they could side-step policy which is meant to preserve correspondence.

    There is software designed to help with FOI requests, but if code words are used, such as turd blossom, your search needs to be wide enough to capture what you’re looking for. And there are document management systems, but I’m not sure they are as prevalent as we might wish.

    Is there legalese for – you know what I’m asking for, don’t use semantic parsing to avoid answering my request?

  22. earlofhuntingdon says:

    Pity that the WH lawyers identified that violations of the Presidential Records Act carry no practical penalty. Kinda like a CIA officer having to pick between obstruction of justice and war crimes. The next administration needs a special “flying squad”, a central office through which suspected administration crimes can be coordinated. Odds are, on top of a few high-profile cases and a score of lesser crimes, that coordination may disclose patterns that ultimately reveal a few WHOPPERs.

  23. Leen says:

    That’s sure a lot of missing e-mails. 5-10 million? Did I read that right?

    tonight it feels like the two war candidates won. McCain/Clinton. Will be saying my prayers for the Iranian people tonight. The Neo-cons are still determined.

    • MadDog says:

      Since they all got “deputized” under Junya’s warrantless eavesdropping program, they’re getting really hyped with the tin badges, water pistols and cans of air freshner, and it’s all gone to their heads.

      Shorter ISPs: “We be Ociffers of the Law now. Let’s arrest some customers.”

  24. PetePierce says:

    I noticed that George Tenant has hired Howard Shapiro to represent him in discarded tape litigation/questioning. If I understand correctly, CIA does not pay legal expenses for them, but many of them are encouraged to buy insurance for legal fees.

  25. JodiDog says:

    This is how conspiracies keep going.

    For example, Mr Fitzgerald had total possession of Mr Rove’s hard drives, etc. (Witness Mr Rove’s lack of participation in a trial.)

    I agree with MadDog. ITs, Engineers, Physicists, Mathematicians and other Scientists and technologists have great senses of humor because we have to deal with the clueless folks of the world all the time, and sometimes we work for them. They just might not understand our humor.

    This is just the never ending search for some kind of smoking gun by the Progressives.

    Consider these possibilities.

    1. There are no incriminating emails.
    2. Or if there had been, the people who had them had the brains to eradicate them well before they were ever looked for by anyone.
    3. Your Government is just not very competent. For example we thought Iraq might be developing an atomic bomb just a few years ago.

  26. JodiDog says:

    Oh yes, just for information.

    This is really just too amusing.

    From above, a quote:

    “Rove voluntarily allowed investigators in the Plame case to review his laptop and copy the entire hard drive, from which investigators could have recovered even deleted e-mails, Luskin said.”
    [[the bold is mine}}

    I don’t know who this guy is, but he is full of it, making such a positive statement.
    I would go along with something like ~might have recovered~.

  27. bmaz says:

    Republicans are the masters of the witch hunt. Democrats, on the other hand, know where the witches are; its just hard to catch the evasive greased up Republican pigs.

  28. Leen says:

    OT…Justin Raimando hits the nail on the head…again.

    http://www.antiwar.com/justin/?articleid=12176

    anuary 9, 2008
    ‘A Heartbeat Away’ From War
    With Iran and Pakistan
    by Justin Raimondo

    As the American people amuse themselves with the illusion that they have any say in the way they are presently governed, our rulers are moving toward war. Two recent incidents underscore the imminence of this prospect.

    The Iranian “provocation” in the straits of Hormuz has set the stage for a new “crisis” manufactured wholly by the War Party, the rationale for which is uncritically accepted by our passive “mainstream” media. We are expected to believe that five minuscule speedboats “menaced” the USS Hopper, a destroyer armed with missiles; the cruiser USS Port Royal; and the USS Ingraham, a frigate. That’s rather like five gnats “menacing” a trio of elephants. Oh, but that’s not all. In addition to intercepting the American flotilla, CNN reports the Iranians supposedly issued explicit threats:

  29. jdmckay says:

    with regards to technical capabilities of creating/managing/searching email archives…

    Update: MadDog, who knows a thing or two about computers, says the backup tapes would most likely not be labeled (that is 1 and 2 would be “no”).

    I’ve done a lot of this work… written several email servers from scratch in early 90’s, and lot’s more specialty email stuff for all kinds of apps (HIPAA >> patient secure email f:ex).

    First & foremost, each mail server has directory of email accounts/addresses/owner(user). Would include also date account(s) added & deleted. All that would be standard for any of the many fine email servers in use currently. This data available independently of each email communication… eg. it’s stored in email server datastore and part of “guts” that email server needs for proper functioning.

    Aside from “contents” of a given email communication, most servers keep transaction info in separate datastore: eg. orgin/destination “address” & email account ID (owner/user), timestamps etc. Maintaining these “transaction records” is standard practice. Most larger ISPs do it now.

    The individual mails, at minimum, are stored w/headers from each communication intact: eg. this is same data you see in your email client (mail program) when examining a “full” header (eg: destination address/server ID, sent & delivered timestamp, “to” & “from” ID, etc.). All this is very easily searched and categorized, whether from “live” storage or archives. There’s a gazillion tools for lexical analysis of headers & body (eg: message text)… regardless of platform (UNIX, Windows, Linux) or server (MS Exchange, Webmail etc.). Huge body of OpenSource mail servers in use, all capable of these tasks.

    The mail contents can be stored w/indexing specificity anywhere from -none- to highly detailed. They can be stored w/each piece of header information (to/from/timestamp(s) etc.) in separate database fields for faster searching, or bulk standard eml formats (all header info in one field). The only difference in searching/parsing various storage structure schemes would be speed: the more detail in datastore (database) organization, the faster the search.

    Regardless, whether from “live” datastore or backups, searching/analyzing/parsing contents is a simple, straightforward matter even for mid-level admins & coders. A decently equipped desktop PC could do a thorough summary through a terrabyte of email records in a couple hours. A moderately powered backup server (machine, not software) will restore 1gb of email backup in +/- 6 mins.

    In this day & age, there’s -0- excuse/reason for any failure/”loss”/corruption… such occurences on MS Exchange w/recomended bakup hardware (eg: redundant storage) are less than .01%, and even those failures can be identified as to cause and easily corrected. A number of UNIX/LINUX mail servers even more dependable… several of the most widely used are free (open source) and performance/reliability is well documented on SourceForge through 1000s of heavy load installations.

    An administrator would know immediately if records were not being properly archived. In most moderate to heavy load systems, both “live” and archived data is “stored” on the fly (eg: real time… when transaction occurs). A moderately secure system would archive individual communications daily during off hours… weekly at worst.

    Loss of email records, as it appears WH has claimed, just doesn’t happen any more. Losing a week’s worth of records would constitute incompetent administration. Any of 100s (1000s?) competent IT experts in this area could testify and conclusvely demonstrate why.

    Years of data loss WH is suggesting is either a lie, or deliberately implemented… or both.

    • bmaz says:

      Years of data loss WH is suggesting is either a lie, or deliberately implemented… or both.

      As with Phred above, thank you jdmckay. And again, no shit! I vote both. This is all beyond the sublimely insane.

    • Rayne says:

      Agree with most of what you say.

      Believe there were at least a couple other DKos diaries that dissected the backups/missing emails, too, cannot remember the diarist’s name. There were some meaty and pertinent comments in them. I trust “wanderindiana” implicitly on this, since wanderindiana also did the legwork to show that a contractor had access to both Republican files and congressional files, was operating inside the firewall.

      There’s documentation that specifies how the archiving was to take place; there absolutely would have been a written methodology for archiving and a network plan that laid out physical redundancies of data. That’s another part of this equation that hasn’t been discussed; likely there was more than one archived copy.

      As an IT tech responsible for backups, I did differentials every day and full back ups on a less frequent basis. But at any time I had multiple full backups in a vault that were taken over the course of a month, and I would also have year-end and month-end and quarter-end backups.

      For example: let’s say the organization only requires a FULL backup once a week, after hours so as not to bog down system and network performance. It gets run each Friday night. By the end of the month, I have 4 full backups; the only difference between this week’s and last week’s back up is the emails sent that week. If the system runs differentials every day, they only back up each night after hours the new content created that day — but you can see where by Friday there should be 5+ copies of the emails sent on Monday.

      GAO documents indicate that there was an effort to harden systems and improve business continuity in case of a disaster, in the wake of 9/11. There should have been increased redundancy, not less, and more widely distributed redundancy, not less, after 9/11.

      Screw the labels; they only indicate which server and whether diff/full, what date, time the backups were run.

      I want to know if the 5 million number is original emails, or if this is ALL redundant copies.

      And if ALL redundant copies are gone, this is a pointed and deliberate obstruction of justice.

    • Rayne says:

      The additional insult here is that this is a small-to-medium enterprise. Not a large or even global enterprise we’re talking about here, with multiple data centers containing multiple petabytes of server farms.

      This is a smallish, manufacturing plant-sized operation, for which management and redundancy should have been a snap. It’s not likely it was lumped in with DoD or DoS or other organizations for security reasons, let alone for disaster recovery and business continuity purposes.

      There’s simply no reasonable excuse we should still be wondering where the emails are, no good reason why we aren’t reading them now.

      • jdmckay says:

        Fully agree… how many WH employees, 1k/2k? Add in Pentagon, maybe double or triple that? Managing 100k such accounts reliably is straightforward, well established practice. this whole “oops, lost the archives” thingie is 100% bull shit.

        It’s a feature, not a bug.

  30. Rayne says:

    Oh, one more point — somebody mentioned Exchange up thread. I swear I saw something that indicated there was a migration between platforms, sometime between 2001 and 2005.

    It’s not impossible that the NEW platform didn’t contain a full migration of all emails, assuming that’s what happened. But I cannot imagine that any IT folks worth their salt would have migrated without having multiple full backups of the old system on hand, and possibly even run the systems in parallel in case a cut-over was needed, prior to ultimate cut-over to a new system.

  31. bmaz says:

    I found this list, which I estimate to be well under 500 people. Check out all the names; this is one inbred group of wingnuts. The level of total nepotism among friends and family is beyond belief. This name caught my attention:

    Mitnick, John M Associate Counsel to the President

    Is he related to Kevin Mitnick? If he is, there is one source of potential computer tomfoolery.

    • phred says:

      Nice list you found there bmaz. Did you notice how many “Record Management Analysts” were listed? One wonders what they do all day…

      Also good catch on the name Mitnick. I wonder how common a name that is…

  32. bmaz says:

    I am sure the odds are against them being related; but out of the population of Mitnicks out there, it does have to be a smaller subset that has the IQ genes necessary to produce these two Mitnicks, so I figured it was at least worth asking….

    It was also amazing how many names were so familiar and are clearly the sons, daughters, sisters, brothers, wives and husbands of others in the cabal. Like I said, inbred wingnuts…

  33. marksb says:

    Somebody needs to send this thread over to Waxman’s office. I’d love to see jdmckay testify to the Oversight Committee…Clear, cold facts to counter the BS from the WH. Nobody in the media is going to ask these questions or spell out why it’s bullshit. Grr!

  34. JohnLopresti says:

    Nice to see the Millenium gear updates in this thread. Which offices among the 2k+ employees in the WhiteHous would still have its own workstations each individually equipped with 1/4-inch tape cartridge backups, likely few, if any, in 2003; probably those desktop devices were replaced at the same time as all the sans-W keycap keyboards in 2000. I like the jukebox mention, though that may be fancy gear for offices of president and veep. When I last had to use a taping system for a workgroup, the software had an arcane userInterface; however, some manufacturers were merging and acquisitioning, and some of the administrator console interfaces were as simple as looking at one’s own folders on a plain vanilla hard drive, and even a restore operation in some of those environments was as fast and easy as replacing a folder or even a file; though, email has other efficiencies less analogous to spreadsheets and word processing documents. I would expect, however, that in our own era IT in the WH and State would be very interesting with respect to keeping as much as possible accessible, and keeping the UI simple enough so workstation ’seat’ occupants could effect their own ‘restores’ without engaging helpdesk for assistance. I agree with the techs in the thread who expostulate onomatopoeiacally at the concept ‘emails could be lost because they were erased completely for those two or three years’. I could see the furniture subcontractor who did IT arranging for a scrub of select emails but not all for two years; and if the subsidiary of any of the principal network gear vendors had lost all those years’ emails, it would explain banning from bid lists in future rfp’s; too outlandish to contemplate and unlikely. I think there is a subscript ongoing in the current lawsuit, something like national archivist seeing the merits in crew’s more politicized, yet curious, intent; as in, presidential records act obligations as critiqued long after the current administration has left office. As this involves the central IT works at one of the two major political parties, there is concern, surely, to preserve and protect the vigor of that party, Republican, for the “offsite” accounts. Another rumination that occurs to me is the tension between ISPs and the Republicans, though many pipe providers are Republican; so, if the provider is archiving, that complicates eradication efforts by Republican server administrators. I think the archivist is protecting the archive’s integrity in this effort.

    That said, though, on a separate OffTopic forTheThread matter, I found an InspectorGeneral report from a part of the archivist’s department in the matter of purloined JGRoberts Reagan memos from the presidential library; amazing: the IG report finally delivered a year later in 2006 80pp long has dozens of REDACTIONS/page. And this is the matter of the nominee to ChiefJ of ScourtUS.

    Forensically, I would wonder if there is time this week for an AndyCard, you have 36 hours before we tell you we want those documents, interlude, in which the missing years’ emails could be located by header but selectively multiply wiped, and if that disappearance effort itself would be documented.

    • PetePierce says:

      John–

      Thanks for the link you provided me the other day. I think by the time I got back to the thread you would probably have left.

      I appreciate it.

    • jdmckay says:

      As this involves the central IT works at one of the two major political parties, there is concern, surely, to preserve and protect the vigor of that party, Republican, for the “offsite” accounts. Another rumination that occurs to me is the tension between ISPs and the Republicans, though many pipe providers are Republican; so, if the provider is archiving, that complicates eradication efforts by Republican server administrators.

      WH/RNC hosts were repubs. And they even keep GOP administered state elections (Ohio ‘04) all in the same family. ePluribus did a nice family tree, more here.

  35. JohnLopresti says:

    P, I am busy primarily elsewhere, true. I think Melissa and company have learned a lot from the secret rollodex foia embroglio and in tandem now with the archivist are on a promising venture in the emails unscrubbing request. It is too long since I toured mid range tradeshows to be precise in the current thread; and my work customarily was second tier, though with moments of something more principal. I still have to finish studying the hundreds of redactions in the Chief Justice pixDust scandal, though appreciate MemHole and wayBack for making that search productive. The recent Nixon YorbaLinda PresidentialArchive cooling off period’s expiration also could be a promising development. There are some good researchers on that history.

  36. Rayne says:

    jdmckay –

    re: (84) — yup, depending on architecture, should be completely automated process and at least RAID-5 on Exchange server, let alone other redundancies they would surely have (my experience in a Fortune 100 enterprise dealt with (3) global data centers so all redundancies wouldn’t exist in one state, country or one continent, but since this is the gov’t, they probably have at least two in 2-3 state radius…and then some).

    re: (85) — I tend to think that there are some segregations, that EOP wouldn’t be with DOD or DOS or other departments, let alone DOD on one (again, having worked in IT for a vendor to DOD, I know they had entirely different services and contracts depending on arm of service). Still, 1K/2K is nothing. Piece of cake to handle backups.

    re: (101) — yes, thanks for that, forgot that luaptifer was the lead on that; wanderindiana was one of the team that contributed to that ePM investigation. Which is still not done, I will interject; there are tidbits that pop up from time to time that lead us back into the same rat’s nest.

    • jdmckay says:

      re your 86: yes, normally I’d agree.

      In my mind (speculation) I lumped ‘em together because of politicization/propaganda from Feith’s shop for pre-Iraq-invasion. Common sense tells me communication between OSP & WH was constant. Also tells me that any records of those communications would send those involved to the gallows. Having a one-stop auto-delete & disk-wiping hosting service would be… convenient. And finding someone to do that job might not be so easy.

      Just my (currently) US $.0167 (and dropping fast) worth speculation.

  37. JodiDog says:

    Emails, emails, emails.
    Torture tapes, torture tapes.
    Plame, Plame, Plame.
    AG, AG, AG.

    Democrats seem to to be looking for things a lot.

    Well there is one exception, a guy knew where the incriminating stuff was.

    Sandy Berger, former Clinton NSA, forefeited his law license rather than tell what he had stolen from the National Archives.

Comments are closed.