Fitzgerald Subpoenas and Server Crashes
CREW reports something that I demonstrated clearly some time ago: materials subpoenaed by Patrick Fitzgerald in his CIA Leak Investigation were lost in the White House’s seemingly intractable problems with email. (h/t Laura Rozen) But CREW’s got new documents proving the case, including this Microsoft Post-Mortem documenting its efforts to conduct an email search in February 2004, in what is almost certainly the series of subpoenas Fitz issued shortly after taking over the investigation (the date referred to in the post-mortem–January 22–is the date Fitz issued his subpoenas). Here’s a summary of the key subpoenaed material:
February 6 was Abu Gonzales’ deadline to turn third batch of documents over to DOJ, including "records on administration contacts with more than two dozen journalists and news media outlets." The journalists, with my best take of what the investigators were looking for in brackets, include:
Robert Novak, "Crossfire," "Capital Gang" and the Chicago Sun-Times [duh!]
Knut Royce and Timothy M. Phelps, Newsday [source for their confirmation of Plame’s status]
Walter Pincus [Libby conversation, July 12 Plame conversation], Richard Leiby {background for profile], Mike Allen [identity of SAO], Dana Priest [identity of SAO] and Glenn Kessler [conversation with Libby], The Washington Post
Matthew Cooper [duh!], John Dickerson [possible additional source, Ari’s "walk-up" conversation], Massimo Calabresi [possible additional source, Wilson interview], Michael Duffy [earlier article] and James Carney [earlier article], Time magazine
Evan Thomas, Newsweek [why Evan Thomas? was he the "they’re coming after you" journalist?]
Andrea Mitchell [see Tom Maguire], "Meet the Press," NBC
Chris Matthews ["your wife is fair game"], "Hardball," MSNBC
Tim Russert [Libby complaint], Campbell Brown [why Campbell Brown?], NBC
Nicholas D. Kristof [Wilson column], David E. Sanger [January 24 document leak] and Judith Miller [duh!], The New York Times
Greg Hitt and Paul Gigot [July 17 NIE leak], The Wall Street Journal
John Solomon, The Associated Press [why John Solomon?]
Jeff Gannon [I knew Plame’s identity from slumber parties at the White House], Talon NewsNote two journalists who don’t appear on this list: Clifford May, who in Fall 2003 claimed to have known of Plame’s identity, and David Cloud, who in October 2003 published an article that appeared to be based on a leak of the INR memo. While I presume Cloud may not have been included because his article was outside the scope of the investigation, I assume May was excluded because the FBI determined in Fall 2003 that he was full of shit.
On March 5, a month past its deadline, the White House agreed to turn over documents subpoenaed on January 22 and due on February 6. Scotty spun the delay thusly:
Q Can you also confirm that Air Force One documents — been handed over to a federal grand jury?
MR. McCLELLAN: Well, I would just say that we are, at the direction of the President, cooperating fully with those who are leading the investigation. We are complying with every request, and we will continue to comply fully with the requests from those who are leading this investigation. No one wants to the bottom of it more than the President of the United States.
Q So they were handed over?
MR. McCLELLAN: Well, we did send — the White House Counsel’s Office did send a letter out to White House staff, urging everybody to comply fully with the request from the investigators, and that’s exactly what we are doing. But, yes, at this point we’re still in the process of complying fully with those requests. We have provided the Department of Justice investigators with much of the information and we’re continuing to provide them with additional information and comply fully with the request for information.
A curious notion of cooperation the WH has there. I guess it isn’t a deadline if the President misses it?
Remarkably (well, not really), the day before this subpoena was due, the White House’s email server crashed:
On 2/5/04, the EOPRM server’s drives crashed and caused the server to bluescreen at bootup.
Microsoft concluded its investigation of the crash with this wisdom.
There was an Exchange database backup but it was two weeks old. If we were unable to access the databases on the bad drives, two weeks worth of query results could have been lost.
Though I’m not sure whether that means two weeks were lost, or could have been.
We know a bunch of the emails in response were printed out by OVP RM on February 11.
February 11, 2004: Date on which June 11, 2003 Martin to Mayfield email printed out. The email was apparently discovered in a search of OVP files by "OVP RM." It mentions "Pincus" and "Niger."
February 11, 2004: Date on which July 11, 2003 Martin and Cooper email exchange printed out. The email was apparently discovered in search of OVP files by "OVP RM." It mentions "Cooper" and "Niger." Cooper’s initial email was printed out, probably on July 11 or 12, though it has no date; Libby wrote notes on it on how he would respond to Cooper.
And many of the days leading up to Microsoft’s efforts to fix the problem were still missing emails in 2005.
January 29, 2004: No email archive of OVP or WH emails.
January 30, 2004: No email archive of OVP emails.
January 31, 2004: No email archive of OVP emails.
February 1, 2004: No email archive of WH emails.
February 2, 2004: Addington drafts a letter to Keith Roberts, Acting General Counsel, Office of Administration, listing the new terms for a search of the OVP domain. If "Joe Wilson" or "Niger" were mentioned in the October 1 gaggle, the October 1 Martin to Mayfield email should have been found in this search. No email archive of WH emails.
February 3, 2004: No email archive of WH emails.
February 7, 2004: No email archive of OVP or WH emails.
February 8, 2004: No email archive of OVP or WH emails.
What a bunch of coinkydinks, huh?
There are more documents relating to OVP that I’m sure will keep me, William Ockham, and MadDog busy all day.
The Office of the Vice President (OVP) had particular problems with missing emails. Several of the documents discuss copying and conducting a manual review of more than 200 PST files from OVP. OAP00000377, OAP00000741, OAP00000790, OAP00001411, OAP00001415. This review may have led to the creation of a spreadsheet compiling information about the OVP PST files, which showed gaps in the dates of preserved messages preserved. OAP00000778. One of those gaps was from at least October 1-3, 2003, a period for which Special Counsel Patrick Fitzgerald sought email during his investigation of the leaks that led the disclosure of Valerie Plame’s identity. OAP00000377.
But Marcy, we can’t be looking backward, President Obama says we have to look forward. Grrrrrrrr.
This is interesting, from the Microsoft post-mortem:
27,920 is a lot of discussions with those journalists.
Oh goody, technical fun and games!! The MSFT postmortem is already a veritable festival of laughs!!
Why? because there was no effort to do a real root-cause analysis based on forensics, only an attempt to make it look like they were recovering the emails.
They were already late on the subpoena.
Note, though, that in October 2003, they moved to a system where everyone kept their emails in their own mailboxes. Meaning Scooter and Karl had access to this stuff for months after the investigation was announced (no wonder Karl had to find his email to Hadley on his own computer). Any bets that’s when they started taping over backups?
Oh, I suspect the “keeping emails in their own boxes” was as much a feature, not a bug, as this was:
This is just two forms of compartmentalization.
There were also other contract firms involved in WH email system which I’ve never heard of; you’d expect to see names like MSFT and Unisys and IBM and so on involved, but who the hell is this?
Which means this firm’s contract BEGAN in 2005…who had the contract before them for whatever this architecture component might have been?
Read the rest of this article for more; the IT industry is highly skeptical of anything the EOP has said about the email system and archiving.
I’ll dig around in my attic for more on taping, not certain if I squirreled something away or not. Right about now I’d kill for EW’s holographic memory…
You might find some hints about the tape operations in this document(PDF), responses of Steven McDevitt to Rep. Waxman and Oversight Committee. See pages 24, 26 in particular.
I see the System Management Engineering firm mentioned in same doc as a vendor, but directly below MZM. Wonder if SME picked up after MZM left off, although the description of their work is entirely different.
An undergrad on CS101 might receiving a failing grade for botching a similar assignment.
I got some “I-T” friends who might be able to sort this stuff out. They can translate hexidecimal by sight
okay, let’s call em what they are
hackers
is anybody considering that angle???
I got buddies who can do things with computers that you don’t want to know about …
ever heard Macitosh’s “Chimes of Death” ???
I have, right before my friend started buggin me for a paper clip …
btw, some of you folks might want to hang on to those old hard drives. jes sayin, is all
If y’all get your hands on one of these drives, even if they have been written over, I’ll get a
hacker I knowfriend to look at it … it might cost us 3-4 Bags of Cheetos though !Unless you have certain logging features enabled (and the default for all of them is OFF), M$ crapware leaves almost no trace of what might have happened. And it’s common to leave them off unless you’re having a repeating problem as they take up a lot of CPU and other such resources.
Boxturtle (I heard a tech gripe about 60% overhead, dunno if it’s true)
Speaking from experience, M$ Outlook does crap like the above. I’m not saying the above “just happened”, what I am saying is that it will be almost impossible to prove that it was not simply M$ crapware deciding to be evil.
Unless we get a human to step up and say something else happened.
Boxturtle (Where I worked, we took backups daily and kept a journal)
yet another ball of yarns to unwind.
i’d love to see someone with the power to put folks under oath have feet-to-the-fire chats with some of the whitehouse data keepers.
these folk might have a lot less to be loyal about than scooter, et al.
but then the current comity-loving white house would probably nix any such effort.
This is intersting, too. Document 741 shows they started efforts to reconstruct OVP emails on October 21, 2005–a week before Libby was indicted.
So Fitz must have gone back to look for more emails after Luskin did his (successful) last minute head fake by revealing the Hadley email.
did microsoft determine what CAUSED the crash ???
was it a computer glitch, of a deliberate act by a human being ???
I ask this looking toward investigating the possibility of a criminal act of destruction of evidence
well, several criminal acts, actually …
See my #7.
And where were you this morning? You could have had blackened Redfish for breakfast.
Boxturtle (M$ software will not rat itself out unless specifically told to in advance)
I was doin turtle research (ya, that’s the ticket)
can’t a guy take some time off to spend with his turtles ???
did ya know turtles fart ???
at least, I’m assuming that was the cause of that foam in the water bowl behind Squirt this morning …
I been on pain medication lately, and it really screwed up my sleeping non-pattern; I been sleeping at night an stuff. And spring has sprung while I wasn’t paying attention, so my flowerbeds look like shit. Add it all up, and it means that lately I been spending more time than usual in the meat world
btw, I straightened redfish out, but I ain’t sure he’s a full time troll
and YES, I changed their water afterwards …
Ok, just saw this. Looking at it now. I expect to find some significant information ‘between the lines’ in Microsoft’s post-mortem.
Oh this is nice.
Fitz subpoenas extra journalists on January 22 (AGAG takes a day to release the request, meaning it goes out to OVP on January 23). January 23 is the last backup before the email server crash on February 4. Just another coinkydink, though, I’m sure.
When was it that Fitz found out that Bob Woodward knew about Plame’s identity and job title? Remember how Bob Woodward was out there filling the air waves with claims that the Plame outing investigation was “much ado about nothing”
Here’s another interesting detail. Document 377 shows a gap that I either hadn’t noticed or wasn’t revealed before: September 16, 2005 through September 26–or most of the days between Libby’s Aspen letter to Judy and the day she testified.
What’s with the two different kinds of redaction in that doc, too, I wonder?
You got me. It looks like the whiteout redactions are to hide the file structure of WH computers. Perhaps NARA did that. And the blackouts are–perhaps–Obama redactions?
Makes some sense, except for the blackout before /SIS as I don’t think that’s a name but a function. Does not make sense for names to be blacked out if any of these people have testified before Congress or provided content at request of Congress, so I wonder about the excess of the blackout?
The answer to this should be interesting because it smells like a common thread that is being suppressed.
There may be something bigger than even we suspect, or simply other people hiding in the shadows that got overlooked (Chimpy or Chimpy Sr.?).
Wait a sec. 2005?! After JudyJudyJudy went to the clink?! They were STILL losing e-mails?! Oh, come ON.
Yeah, specifically for the period between when Libby wrote her a letter telling her Dick Cheney was sucking her roots and just a few days before she agreed to testify. I would imagine they were in a panic on those days.
(fp; ya otta try MY hours if you want a screwed up night and day…work 3p to 1a)
I can write a simple snippet of code to crash a server if I have access to it, and I’m not even a system level programmer; I have done it for server hardware testing.
Sorry for my OT hit and run:
Supreme Court rejects right for criminals to DNA tests. It seem they are “worried” about a flood of cases from those nasty innocent people supporting our for-profit justice system.
Maybe we all otta’ move to Iran where it’s civilized.
Control back to the IT savy…..
27,920?
Were the journalists in the White House using White House computers to correspond with the White House?
If not, they they ought to have copies of the correspondence on their respective computers.
Why couldn’t we ask the journalists for copies they might have?
(or as someone else mentioned in a prior thread, why not ask NSA to cough up a database or two and direct us to the missing emails)
[Edit] When I write “we” I mean Fitz or someone similar with investigative power.
27,920 “comprised a query result” – may not mean these were all fully responsive, could have been interdepartmental idle chatter, or missives back and forth with RNC about handling the political stuff related to same.
After seeing Whitehouse’s chart of communications contacts between WH-EOP and DOJ under Bush admin, wouldn’t be surprised to see a crapload of chatter across the organization. Very undisciplined people.
thanks, that helps some.
For such undisciplined people, I have to admire how well they do as a group to keep quiet, however, when there is an ongoing investigation. crickets.
unless all their chatter was about what they thought Gannon had done last night or somesuch.
This reminds me, did the MSM ever get around to looking into Jeff Gannon?
Way O/T, and with apologies, because this thing with the emails is one hot topic, but here’s another from the late great ‘68. Still depraved after all these years.
http://rawstory.com/08/news/20…..riot-cops/
I’ll have more to say later,but this is just bizarre:
They got a subpoena in October 2003. They needed Microsoft’s help to answer it. At the same time, they changed their process which ensured that Microsoft’s solution wouldn’t work the next time and didn’t tell Microsoft. I just can’t believe that was an innocent mistake.
Oh sure. It was not an innocent mistke. But does that make it bizarre?
Though one question is whether they needed to invet a second reason to “lose” email responsive to that subpoena?
That is, if they did one thing to “lose” their Novak emails subpoenaed (actually, requested) in October 2003, and then were subpoenaed again, would they need to invent a second reason to “lose” them?
rubber emails ???
ya just can’t throw the fucking things away, they keep popping back out of the trash
from what I been told (and what I’ve seen some crazy people do) if you wrote it on a computer, it’s still in there, some where
one of the people I got in mind was once described aptly as a “Loose Cannon” by the computer administrator where I was working
I say we let the “Loose Cannons” have a go at it
they can’t fuck it up any more than it already is
and they can probably tell us what is on that hard drive …
Sorry, I left out some context. Let me outline what happened (like any good consultant Microsoft carefully documents the client’s screwups, but in a very non-obvious way).
Before Oct. 2003 – No Exchange email was being copied to ARMS. Instead, it was pulled out of journal mailboxes into PST files. It’s not yet clear to me whether the jury-rigged Legato to custom code to ARMS solution was being developed or was developed after the fact. In any event, they called Microsoft in to find the emails because they had no way at that time to find them.
After Microsoft left in Oct. 2003, somebody had the bright idea of leaving all the email in the journal files (while waiting for the new archiving solution). Apparently, the idea was that they would search through the journal files for each agency when they got a subpoena.
Jan. 22, 2004 – they get the subpoena. They spend two weeks doing searches. They use special Records Management accounts housed in an Exchange Server called EOPRM to do these searches.
Feb. 5, 2004 – The day before they are supposed to return the documents, the hard drives (note the plural) on this Exchange Server crash. All the search results disappear. The called Microsoft back in to fix the problem with that Exchange Server but they didn’t tell Microsoft why they needed it restored. Microsoft builds a new Exchange Server and successfully restores the Exchange database from the bad drives (this was either near superhuman effort or the drives didn’t really crash).
Feb. 10, 2004 – they call Microsoft because they’re having problems completing the searchs of the gigantic journal mailboxes. Microsoft tells the EOP that the results from WHO (i.e. Karl Rove’s mail) were potentially incomplete. I suspect that the 27,000+ results were returned from the WHO journal.
Gotta run to a doctor’s appt. Back later with more.
Still waiting for those lost e-mails after all these years.
at the risk of sounding pendantic, your hard drive has probably crashed / malfunctioned if you start up the computer and get an error message stating “Operating System not found.” If you get a ‘Blue Screen of Death,’ it means Windows has at least partially loaded, but malfunctioned. meaning the hard drive disk was functional a few seconds ago.
now, i don’t know about maddog and WO, but the first thing i do when a client’s machine fails this way, is to remove the drive, plug it into another pc and recover the data i was after.
It never fails – so far. but then, I get paid for keeping the data, not deep-sixing it. If Fitz could get the actual, physical drive or drives, any competent computer science major could recover the emails.
so i don’t buy the proposition that the emails are irretrievably lost. they’re just hunkered down, waiting till the GOOPers go away. your data is afraid of criminals, doncha know?
IF we could get ahold of an individuals laptop, we might have something. The outlook client app has autobackup that is normally on and on a laptop the default is the internal hard drive, not something on the server.
But that’s a pipe dream. As soon as the emails got lost, I’ll bet somebody went in and cleaned up any computer that might have revelent .pst files.
IF there was actual interest in investigating this, the original laptops could be located and recovery attempted. Or more likely, we’d find the laptops cleaned of info from that exact time period. Perhaps someone would start to suspect something if ALL the archives failed for the exact time period on multiple machines.
Boxturtle (But likely not. Obama clearly wants BushCo wrongdoing to simply go away)
Does anyone find it disconerting that the Bush WH did a sole source contract with the guys currently doing the Records Manamgenemt?
Actually, I think the Obama White House chose EMC. The answer to your question is no, EMC is capable of doing excellent work. They are a huge improvement over the Documentum solution put together by Booz Allen (which never worked, but the contract documents are there in the middle of this mess of documents).
No, not when it comes to EMC. At the time EMC was still the most powerful kid on the block when it came to certain kinds of storage methodologies.
It’s the reason they were sole-sourced that’s a bigger problem. Why was Booz Allen Hamilton the firm used to vet firms for use by the EOP-OA IT? Why was an outside contractor used for this purpose at all, instead of the EOP-OA management team?
I’m equally concerned about what system the Obama WH is using instead. Bush left a dog’s breakfast of a WH telecomms system, presumably intentionally, so Obama would have had expensively to start over.
Time aligning the missing email periods from the MS post mortem with the subpeonas already shows too many covenient lapses for an honest sniff test. It also reveals a distinct lack of understanding by the contractors of their most basic responsibilities. Nobody is that dumb that often.
Squeeze the geeks. They are in on the gag and if they aren’t retired from all the bags of money they made confounding Exchange backups like this then they have valuable evidence of wrongdoing.
PST’s generated by EXMerge.exe will go anywhere quickly. CD, DVD, thumbdrive, FTP, whatever…
Don’t go after the managers, go after the technicians.
in response to “Don’t go after the managers, go after the technicians.”
and let’s do so before they are all eaten by airplanes.
Like you, I immediately thought of Mike Connell, Rove’s computer guru. I’ve searched, found many articles, all but one implicate him and his variously named companies in the whitehouse e-mail crashes.
The article which yielded the best and most info was HUFFPO’s dated 10/21/2008 by Rebecca Abrahams. It gives Connell’s history of computer work for GHWB in 1988 on through GWB/Cheney/Rove. It lists names of many companies owned wholly or partly by Connell, his wife, or close associates.
Interviews with Stephen Spoonamore, CEO of Cybrinth, LLC, a leading cyber security expert. His 06/12/2007 e-mail to author states among other details:
“…There is no difference between GovTech and SmartTech. Randy Cole and his wife, siblings and a few friends own a rainbow of companies, all privately owned, and constantly changing names.
Spoonamore adds, “Connell has knowledge of where the e-mails were routed and where they may be recovered because he built and maintained the system.”
Spoonamore states that on 10/22/2006 Connell asked him about ways to “permanently destroy hard drives”. Spoonamore said, “If this is what I think you’re talking about, this meeting is over.”
I’m presuming that all here know the story of Rove’s threats to Connell if he didn’t take the fall for the 2004 rigged Ohio presidential election and Connell’s death 12/2008 in the small plane crash 2 days before he was to testify in a court case re that Ohio rigging.
In response to your comment and to WO at (42) –
The problem is that PST files corrupt readily when they exceed 1.6Gb in size, and the methodology by which these guys were backing up their work meant that there were frequent corruptions since they exceeded 2Gb.
The other problem is that the Exchange servers were wide open on the network; stuff could disappear or be encourage to corrupt before or as it went into a PST.
Which makes MZM’s utility in this environment along with other players rather questionable.
Having played IT fireman for situations like this in more than a few places, I can say that this entire scenario of using PST’s as the default Outlook delivery slot makes even less sense if these people regularly exceeded their 2gb limit for mail in a PST. I tend not to believe that they were exceeding 2gb regularly though. That is *a lot* of mail.
The PST2GB.exe utility you use to fix a PST once it does stop responding is also somewhat unpredictable in what it recovers in the truncated PST it creates. You can run it 3 times and get different emails out of the same oversized PST you are fixing. I’ll run it until they say uncle if that is what they need. It conditions them to plan things better when they lose data due to this situation. Especially when it is a federal offense.
That is why I say this doesn’t add up when viewed in hindsight. Multiple archive files created on a schedule and perhaps using the Exchange email store on the server (that’s why you have the server) would be sensible to any rational soul over the implausible scenario we are being sold.
The huge PST files weren’t for people. They were emails pulled out of journal mailboxes for entire departments.
So the link back at 63 indicates tinkering with PST files. These same PST’s? That means someone was in them and massaging them before the robocopying.
Read from the bottom up on that chain. The top email is saying that it was a false alarm.
Question for you: did PST2GB.exe exist in the time frame 2002-2006?
The current version for Exchange2k was published 10/21/2002 and I was already well acquainted with it by then.
MS link http://www.microsoft.com/downl…..laylang=en
“4 servers separated by agency”
Do we know the agency names?
Completely OT, but it made me smile so I’ll pass it along
Site collecting some of the best twitter responses to Hoekstra’s hyperbole on the GOP being just like Iranian protestors.
Grill the techs, thats the ticket, and a lot of reminding of the consequences regarding perjury.
emptywheel:
Were those contacts I sent you a year or so back of any use?
O/T, but exceptionally happy news:
Abousfian Abdelrazik, a Canadian citizen who has been held captive in Sudan since 2003, twice arrested and interrogated/tortured by proxy, declared innocent after all that by Sudan, CSIS, and the RCMP, who is yet still on some perverse U.S. list, is going to be allowed to fly home to Canada.
The Canadian government has moved the goalposts on the guy — who is clearly innocent of everything — so many times, in spite of court rulings, I can’t tell you (’cause I’m too excited). Abdelrazik’s story is one of the more bizarre to come out of the war on terra — he’s been living in the Canadian embassy in Khartoum for more than a year, and yet refused a passport by his own government. Citizens — including a former solicitor general — have bought him return tickets a couple of times, but the rules kept changing.
But the government caved! They gave in! They’ve stopped appealing! I can’t believe it.
You’ve brought a lot of attention to bear on this issue and I am very glad you are finally getting a good response.
Meanwhile back here, I’m trying to process our Senate in a short period of time both voice voting to cover up pictures of American war crimes and prisoner abuse and then issuing a resolution apologizing for slavery.
Okay, so which of you folks is an Exchange/Outlook professional?
What’s “Mail Attender rule was created with incorrect naming values” mean to you?
I’ve seen a previous example of a failed storage where an “incorrect name” was used, “losing” emails, but in a different system other than MSFT. What about MSFT?
Damnitall, let me revise that question: who among you is familiar with both MSFT Outlook/Exchange AND IBM Lotus-related Mail Attender?
I need somebody to confirm in theory that “Mail Attender rule was created with incorrect naming values” could mean that someone set Mail Attender to “DELETE” or other unhappy “incorrect” value “accidentally”, so that journaled PST files were destroyed.
Don’t know what these two email chains mean, but they were done just about the time when the “not archived properly emails” were printed out for Fitz.
Who would have been manually copying to the wrong file on 12/27?
Stinks of Cheney’s Skinks. (nothing against skinks, mind you
Keep up the damn good work, EW, love ya.
It seems to be that “all the lost-email” legal challenges should be consolidated into one big fucking class action suit against Bushie and Cheney.
The jurisprudence of Yogi Berra
“If you ask me anything I don’t know, I’m not going to answer”
And this one reflects the reconstruction process in teh days before Fitz let it slip that emails were missing.
This email chain shows someone discovering unauthorized movement of files around November 10, 2005–OVP is just one of about 10 entities that appeared to be moving files around.
the google says that “Mail Attender for Lotus Notes is one of the most comprehensive mail administration products available.”
I reckon a Lotus delivery rule was formatted incorrectly. I wonder whether Lotus email was not able to honor a delivery rule because it was malformatted, or whether a conversion program that was converting Lotus Notes rules to MS Exchange rules could not read the naming values (either because the names were malformed or because the conversion program had a flaw. The flaw being that it was not designed to recognize all of the possible naming formats.)
Here’s an invoice from “documentum” software, which was put through teh uber-corrupt MMS department at DOI.
Documentum is a records management solution. Some folks like it, but I’ve never been impressed with it. Right after that invoice is the one bumping up Booz Allen Hamilton’s little million dollar project. If I remember correctly, there’s a previously released document that fills in the blanks on what was going on at that time.
It’s not just Documentum, but an order to Symbiont and an amendment to order to Booz Allen included in that document. The network cabling period covered under the Symbiont contract is worth noting: Aug 1, 2003, to Sep 20, 2005.
Hmmm…
And all three of them went through MMS/DOI. WTF???
At the time, MMS/DOI controlled a huge general services contract that allowed agencies to avoid competitive bidding. It’s just a coincidence that MMS/DOI was the most corrupt agency in the federal government…
But don’t you think that helped the staff get cocaine? If you’re the gateway for big contracts?
I think you forgot the /snark tag. Heh.
Mail Attender is a Lotus term as far as I know. If you tell the server to put mail in a place that doesn’t exist, I tend to believe it will try and generate an error to prove it tried.
That doesn’t mean it can or will create a new spot for the email until you figure it out. Another reckless shortcoming, btw.
This seems useful–it’s some of the early math on the gaps in PST files but with some file tracking.
This one may be very useful–it’s got all the OVP files they were trying to replace with filenames and directories (htough they appear tobe the directories for 2005).
Minor OT – Booz was also the “outside auditor” that was used for the SWIFT program that violated European privacy laws and oddly enough, when the European privacy court reviewed the Booz audit efforts, they weren’t all that impressed.
More major OT – Wired has this piece up some cyber efforts to help out Iranian protestors including this:
I guess if our own gov is going to treat us all as if we were Iranian insurgents anyway …
This is a snitty email yelling at SIS for not listing the exact size of PST files (the sizes were getting bigger when they got opened, so size was a critical tracker).
Read the attachment to that snitty email closely. That’s where I noticed the “Mail Attender” problem.
It could well be the very same “error” WO cites at (76) — and something like this has happened before. Another administration had problems with naming conventions at convenient times on emails, resulting in the loss of a sizable number of messages.
Simple naming problems, like using upper or lower case in a definition. Could be seen as a typo, but when it happens repeatedly, jeepers, looks less like a bug than a feature.
The Mail Attender issue is a completely different problem, believe it or not.
Oh, I believe you, there’s dozens of different problems here. Books worth of problems.
My gut tells me Symbiont is a front, a dummy. Their website is cheesy, asks for older version of Java, has a spare look common to other front businesses; they also registered with the government as a payphone provider in October 2003, so I have to wonder if they were not only “wiring” but doing other, um, voice capture services within proximity of the White House.
We still haven’t found the documents pertaining to MZM, have we>?
Not that I’m aware of, but Booz Allen could have been paid amply enough to fund a subcontractor…
And there’s two little firms here that look fishy. Symbiont just smells to high heaven. They are sole-source no-bid, labeled a small-disadvantaged business as well; their president worked for two hotel corporations before moving to Symbiont. Not exactly the kind of portfolio of experience I’d want working on an effing EOP project, you know? I’d have been looking for somebody with government security or military background in this capacity.
Oh, you definitely should look at Symbiont. And then you should look at the firm which is listed as their website developer, Winstar Government Solutions — because both Symbiont and Winstar do “Other ADP and Telecommunications Services (includes data storage on tapes, compact disks, etc.).”
I can’t nail them down, but it wouldn’t hurt to keep these two in mind going forward. Rather interesting that the years in which Symbiont isn’t making $$ from gov’t contracts, Winstar is, and neither of them show contracts in 2000.
And as for “Systems Management and Engineering Inc.” which Steven McDevitt cited as one of the contract firms with which he worked, I can find absolutely nothing in the way of contracts extended to this firm via FedSpending.org. Further, the only reference on the internet to this firm by this exact name is McDevitt’s statement.
The Bush WH supposedly turned in “over 200 pages” of MZM-White House contracts to Waxman’s (former) Committee on April 25, 2007.
Why haven’t those ever seen the light of day?
http://www.warandpiece.com/blogdirs/006024.html
Also, War & Piece has several good pieces on the MZM contracts with the EOP and OA that were sourced through MMS. I go back to them frequently for date checks:
http://www.google.com/search?q…..&sa=2
See also my comment at 89; back in 2006, Laura Rozen was onto to the possibility that MZM was getting contracts through BAH and others, which may mean less traceability than we’d like. Explains why we don’t see quite as much clarity on MZM even though we are getting more and other info.
So I think BAH is the big baddy here, plenty of moola under their contract to pay for others who in turn will provide both the studies to justify their own deployment and then do the work requested.
I have not seen much lately on another computer related story – the plane crash death of Mike Connell, Rove’s IT guy. However, it still looks very suspicious as far a I can tell.
http://adap2k.blogspot.com/200…..nnell.html
Check this out:
[My bold]
These guys have the worst luck. /snark
Gosh, 60 days after 10/11 pretty much runs out any grand jury, doesn’t it?
FWIW, one of the ones in that series is annotated.
Also, correct me if I’m wrong, but the entire executive summary is redacted, correct?
Whtat, does it say, “Plan to Beat Fitz”?
Normally, I would totally buy the idea that someone accidentally renamed the company attribute on all user objects in Active Directory because no one ever does that on purpose. It would be evil. Generally, my rule is never assume malintent where mere incompetence would suffice as an explanation. This is a special case.
Here’s the requirements document for the Exchange to Arms system that didn’t work (the one Microsoft complained about).
Between that and the Microsoft analysis (which includes most of the code for the custom interface), I’m pretty sure I could recreate that application. Not that I would, mind you…
What is UNK?
It’s one set of emails they needed to restore, but I don’t think it’s a component of EOP.
Universal geek abbreviation for Unknown.
Proof I’m not a geek!!!
Thank you–I’ve been working on that since I won all the math and science awards in high school. I hereby declare beer thirty to celebrate my non-geekdom.
Glad to help….
You want to know what sucks? Adobe Reader has a limit to the number of files you can have open at once. And worse, if you exceed the limit, it gets really confused.
Oh yeah, I got halfway through, closed everything, and read through the second half of the batch.
Oh, beer thirty! How’d I miss that? Time to make outdoor grilled ham-and-cheese panini sandwiches for the kids while I quaff a mug.
Raise a glass to Froomkin while we’re at it, hope he gets the last laugh on that dying institution, WaPo.
[edit: Citizen92, thanks for the links on MZM, surfing through them now while I wait for the grill to heat up.]
UNK = unknown ?
This one talks about 30 OVP source filse not accounted for–I think bc they had to be moved.
compare to this one.
That’s just 15 though.
OT Dan Froomkin canned
http://rawstory.com/blog/2009/…..gton-post/
I’m so bummed about Froomkin I can barely read anything.
Hey, I know it’s beer thirty and all. I’m about head out, too. But it occurs to me that we’ll be needing a timeline for this stuff. Especially for Oct – Dec 2005 when the whole place went into a panic about the missing emails.
She’s probably all over it like a tick on a dog, WO, even with a Beamish in her hand.
I’ve culled a few dates, but she’s already got some of them in the existing WH emails timeline; might need to tweak the content attached to the date to flesh out what happened. (This is when version control goes to hell on documents like this, can’t tell going forward what was added when without resorting to the Wayback Machine…)
I updated this with key dates from this document dump–though I may not have understood some of the 2005 stuff well enough to figure out what needs to be included.
http://emptywheel.firedoglake……-timeline/
As discussed yesterday, the “Hate Crimes” legislation is really about limiting speech – with the ADL at the center of preventing the truth from being discussed – under the false flag of “Hate Speech.” Look at the face and the organization that wants to make certain speech illegal:
CBS Editorial Urges Internet Providers Remove “Hate Speech”
excerpt:
“In response to “hate” as defined by the ADL and the Department of Homeland Security, internet users, internet companies, and educators “should speak up” and “counter the vicious lies,” including 9/11 truth “conspiracy theories,” which “have become a core part of the belief system of anti-Semites and millions of others around the world,” according to ADL National Director Abraham H. Foxman. (Note how Foxman links antisemitism and 9/11 truth.)”
http://www.infowars.com/cbs-ed…..te-speech/
I’d just like to point out that Israel is and always has been a country, not a race or a religion. When it spies inside US borders, it’s called espionage. When US citizens do it inside the US on another “country’s” behalf, it’s called Treason.
These are simply facts. No hate in them whatsoever. The truth does not have an anti-Semitic bias. It simply is…the truth.
Cheney’s Plame interview with FBI to be reviewed.
Is Sullivan one of the good guys?
I just stumbled upon that, too, Loo Hoo. Can you imagine the gubmint lawyers making this argument with a straight face?
“Justice Department lawyers told the judge that future presidents and vice presidents may not cooperate with criminal investigations if they know what they say could become available to their political opponents and late-night comics who would ridicule them.”
http://hosted.ap.org/dynamic/s…..TE=DEFAULT
There is no more important national security interest than protecting Obama, Cheney, Biden, Bush et al from Jon Stewart. I can’t believe the DOJ lawyer actually invoked protecting someone from The Daily Show (specifically) as a grounds for keeping their statement to the FBI on the CIA leak “secret”
I can see the new deodorant ad: Secret – strong enough for national security, but made to keep politicians from being ridiculed by The Daily Show.
Loo Hoo – Sullivan has been holding DOJ lawyers to a minimal baseline (emphasis on the base) standard of behavior which, in my book, makes him one of the very good guys. He handed the Steven’s trial sanctions, the GITMO “oops, did we forget to tell your our witness is crazy even AFTER you ordered us to?” situationon the Yemeni detainees, etc.
Maybe we could get Jon Stewart to provide an under seal affidavit to the court that he promises not to make fun of what’s in Cheney’s interview? All better now.
Exquisite humor from you today, Mary. What brand and what dosage of humor pill do you use? Love it!
I dunno, I still think Emmet Sullivan was pretty fucking weak tea on Stevens. He was being lied to his face, said he knew it and still let it go on and on. He should have bounced that case and sanctioned those malefactors long before he did.
Actually, that’s a relief. When Libby’s team tried to bury the leniency letters they submitted for him (no doubt to hide the number of people implicated in the leak who wrote letters of support–which of course should be dismissed outright), they worried about the bloggers (me!!) getting a hold of them. At least here they’re picking on a more dangerous enemy.
Oh, Ms. Wheel, don’t underestimate the amount of worry you cause those who deserve each and every drop of it. We don’t! You’re on the side of righteousness.
woo hoo loo hoo. good news.
Are the clouds over the V.P.’s office going to lift?
Weird. Another little vendor cited by Steven McDevitt is TKC Communications. TKC is part of a holding company structure of small service providers; the parent is NANA Development Corp., which in turn is owned by the Inupiat of Alaska. Very odd range of businesses, from those which are Alaska-based and oriented around natural resources, to a collection of businesses in the DC area which provide a plethora of services to the government.
Passing thought: Where is Ted Stevens these days, I wonder?
Didn’t Toobz or his buddies use some Native American contracts to contrat through on?
You looking for this?
http://dir.salon.com/topics/ted_stevens/
Well, this would certainly fit into the schema whereby members of Appropriations Committees were co-opted, Stevens having been chair of the Senate committee until 2005.
TKC Communications could be legit, but given the stink around the email system, the oddness of this firm’s provenance and the vagueness of work descriptions given for the three smallest firms, my hackles are standing on edge.
I looked for PAC funds from TKC and another affiliate, Qivliq, but didn’t find any; need to look at NANA to see if they made donations.
Thanks, Mary. And fatster, if they don’t want to be laughed at, perhaps they shouldn’t do such laughably stupid things. They might even consider following the law.
Obama, Holder, Leahy, Feingold,Whitehouse, Pelosi “no one is above the law”
Many peasants are doing everything we can and hoping and trying to believe that our Reps and President mean what they keep repeating
WO — you out there? Just had some sort of flashback to my project management days when I had to oversee recovery and migration teams for server farms.
The emails mention using robocopy; isnt’t there a problem with this approach? I thought robocopy couldn’t sweep up all the data associated with a record on a drive. For the life of me I can’t recall what utilities we used instead for data and email servers, but it wasn’t robocopy (think it was an old HP utility we used instead). ??
@119 – it’s a Zantac/Advil cocktail.
@120 – you might find this interesting:
http://www.msnbc.msn.com/id/31….._politics/
Welch was also heading up the Kevin Ring investigation from the Abramoff spinoff (remember how we were going to get so much great stuff from Abramoff?) and has now been pulled.
Jerks
http://www.msnbc.msn.com/id/31…..itol_hill/
The wildly Democratic House has passed a bill, the PMS 2010 (protect my seat in 2010) bill, to make sure that Obama can’t turn GITMO terrorists loose in the US during the 2010 budget year.
Unfreakinbelievable – but he deserves it and every international difficulty that comes with it. Fess up to the truth, as you should have months back, THEN ask Congress for what you need. If you want to play the “terrist terrst terst” game instead of being President, go for it puppy.