Would NSA’s New Big Social Media Data Approach Have Noticed the Arab Spring?

Screen Shot 2014-01-27 at 10.02.29 PMSometime in 2011, I was on a panel with the Democracy Now’s Sharif Kouddous — whose tweeting from Tahrir Square played an important role in keeping the world informed after Hosni Mubarak shut down the Internet. I mentioned that DiFi had been bitching for months because the CIA and other intelligence agencies had missed the Arab Spring.

Who had followed Sharif on Twitter, I asked? (Probably half the rather large room raised their hands.) Because if you had, you knew more about the Arab Spring than the CIA did.

Which is the underlying context to the NBC/Greenwald report that GCHQ collects data from Facebook and YouTube to try to monitor the mood of the world.

The demonstration showed that by using tools including a version of commercially available analytic software called Splunk, GCHQ could extract information from the torrent of electronic data that moves across fiber optic cable and display it graphically on a computer dashboard. The presentation showed that analysts could determine which videos were popular among residents of specific cities, but did not provide information on individual social media users.

The presenters gave an example of their real-time monitoring capability, showing the Americans how they pulled trend information from YouTube, Facebook and blog posts on Feb. 13, 2012, in advance of an anti-government protest in Bahrain the following day.

More than a year prior to the demonstration, in a 2012 annual report, members of Parliament had complained that the U.K.’s intelligence agencies had missed the warning signs of the uprisings that became the Arab Spring of 2011, and had expressed the wish to improve “global” intelligence collection.

During the presentation, according to a note on the documents, the presenters noted for their audience that “Squeaky Dolphin” was not intended for spying on specific people and their internet behavior. The note reads, “Not interested in individuals just broad trends!”

What we’re seeing is how NSA would go about amassing public data to try to learn what the rest of us can read by following Twitter attentively. [see update]

I won’t comment much on the technical ability here (which involve contractors to collect the data), and I’ll only applaud that Facebook has finally been exposed as the perfect surveillance app it is.

But there seem to be several problems with the analysis they’re doing (though MSNBC did not include the script for its PowerPoint). Aside from what seems to be an Orientalism built into the analysis…

Screen Shot 2014-01-27 at 10.29.41 PM

And some half-assed PsychoLOLogy…

Screen Shot 2014-01-27 at 10.32.51 PM

Nowhere does this presentation distinguish between the propaganda social media accounts and the legitimate ones — a known problem of social media analysis going back years (which has, because of the all the competing parties involved, been particularly acute in Syria). Perhaps they deal with this, but this analysis seems ripe for spamming by propaganda, particularly if it came from frenemies who know GCHQ and NSA use such analysis.

Now, presumably someone somewhere else in the combined Intelligence Communities of the US and UK would actually sit down and read the social media of a potential hotspot, which is the way a bunch of Tweeps in their pajamas can get a sense of what’s going on without collecting all the social media data for an entire country first. Such an approach uses the hive mind you acquire on social media, with the built in assurances from trusted interlocutors.

After the Arab Spring, the Intelligence Communities of a number of nations got their asses kicked because none of them are well suited to figure out what non-elites are doing. But from the looks of things, they just hired some contractors with bad attitudes to have something to offer up, no matter how dubiously effective.

Update: My statement was inaccurate. They got this data by tapping the cables.

Why Has the Intelligence Community Missed So Many Digital Bales of Hay?

In a piece on the intelligence community’s increasing reliance on SIGINT, LAT reports that the amount of the President’s daily brief that comes from SIGINT has increased from 60% since 2000.

Determined to identify and track Al Qaeda terrorists and to prevent another attack after Sept. 11, 2001, the NSA set about vastly enlarging its ability to capture, store and exploit the ocean of texts, emails, videos and other electronic communications.

“They took on a new mission that required sifting vast amounts of data to find a few important signals,” said Stewart Baker, who was the NSA’s general counsel from 1992 to 1994 and held top Homeland Security Department jobs in the George W. Bush administration.

Today the NSA secretly siphons an almost unimaginable number of foreign government, corporate and private communications from the World Wide Web, according to the trove of classified material disclosed by Edward Snowden, the fugitive former NSA contractor. One document leaked last week revealed that NSA computers take in 500 million “communications connections” per month in Germany alone.


About 60% of the president’s daily brief, the highly classified intelligence summary delivered to the White House each morning, was based as of 2000 on “signals intelligence,” or intercepted communications, according to a declassified NSA document from December of that year. The NSA portion has increased since then, former officials say.

“Over the last 10 years, because of the Internet gold mine, signals intelligence has become the primary vehicle for U.S. intelligence collection,” said James Lewis, director of the technology and public policy program at the nonpartisan Center for Strategic and International Studies in Washington.

WaPo’s original story on PRISM (which, remember, is just a computer interface making it easier for analysts to access data from just 9 companies) reported that 1 in 7 pieces of intelligence in the PDB derived from PRISM, or a total of 1,477 pieces of intelligence last year (10,339 pieces of intelligence in all the PDBs last year, then?).

An internal presentation of 41 briefing slides on PRISM, dated April 2013 and intended for senior analysts in the NSA’s Signals Intelligence Directorate, described the new tool as the most prolific contributor to the President’s Daily Brief, which cited PRISM data in 1,477 items last year. According to the slides and other supporting materials obtained by The Post, “NSA reporting increasingly relies on PRISM” as its leading source of raw material, accounting for nearly 1 in 7 intelligence reports.

Remember, this is all non-public information.

Back in 2011, however, the intelligence committee failed to understand the Arab Spring that was breaking out in public fora for all the world to see (I once quipped that those who followed Democracy Now’s Sharif Kouddous on Twitter had a better understanding of what was going on than the CIA).

And as recently as this year’s confirmation hearing for John Brennan, he admitted that the CIA needed to better monitor public social networks.

BRENNAN: Well clearly, counterterrorism is going to be a priority area for the intelligence community and for CIA for many years to come. Just like weapons proliferation is as well. Those are enduring challenges. And since 9/11 the CIA has dedicated a lot of effort, and very successfully, they’ve done a tremendous job to mitigate that terrorist threat.

At the same time, though, they do have this responsibility on global coverage. And so, what I need to take a look at is whether or not there has been too much of an emphasis of the CT front. As good as it is, we have to make sure we’re not going to be surprised on the strategic front and some of these other areas, to make sure we’re dedicating the collection capabilities, the operations officers, the all-source analysts, social media, as you said, the — the so-called Arab Spring that swept through the Middle East. It didn’t lend itself to traditional types of — of intelligence collection.

There were things that were happening — happening in a — on a populist — in a populist way, that, you know, having somebody, you know, well positioned somewhere who can provide us information is not going to give us that insight, social media, other types of things. So I want to see if we can expand beyond the sodestra (ph) collection capabilities that have served us very well, and see what else we need to do in order to take into account the changing nature of the global environment right now, the changing nature of the communication systems that exist worldwide.

Though Brennan suggested that a focus on leaders rather than common people led to CIA’s blindness in this case (I’d add, a reliance on brokers like Egypt’s Omar Suleiman or Saudi Arabia’s Mohammed bin Nayef, who have an interest in depicting unrest in their countries as threats to friendly governments, distorts reality).

But whether the NSA or the CIA should have seen the revolts bubbling up in plain sight, both missed it because of all the secret stuff they remained focused on.

I’m not actually advocating for the CIA to start trolling Twitter more aggressively. Still, if the focus on secret stuff has led to blindness, we need to rethink our obsession with secret digital haystacks.

Fresh Allegations of Torture in Bahrain

On a day when President Obama is at least making the admirable move of visiting the West Bank and speaking favorably for Palestinian statehood after his visit to Israel (to lend legitimacy to Netanyahu’s continued desire to attack Iran?), it is easy to overlook a report in the Wall Street Journal in which we see fresh allegations of torture continuing in Bahrain.

Recall that in the aftermath of Bahrain’s brutal crackdown on its citizens trying to join in the Arab Spring movement in early 2011, one of Bahrain’s “reforms” was to hire notorious police thug John Timoney to run its police force and to “implement” the findings of an independent commission that had been brought in to investigate torture and other abuses by the government. Just a few months after taking charge, Timoney took the repressive step of banning all protests while jailing a number of prominent protest figures. A couple of days later, there were mysterious bomb blasts that might well have been the work of Timoney’s known practice of infiltration since they were not directed at government targets as one might expect if they were the work of a developing resistance movement. US actions in response to abuses on the part of Bahrain’s government has been especially lame since the US is so attached to its base for the Fifth Flleet in Bahrain and “security’ for the flow of oil from the region.

The new allegations of torture include torture of suspects arrested for those November 2012 bombings:

Five detainees arrested in Bahrain last year said they were tortured in custody, according to family members, lawyers and an ex-prisoner, accusations that a member of an official inquiry panel said should be formally investigated.

Bahrain security forces used methods including beatings, electrocution and suspension on ropes to force confessions from the detainees, who were accused of involvement in bombings in the capital, Manama, the people alleged to The Wall Street Journal. The Bahrain government said the torture allegations were false.

The claims suggest the Bahrain government has failed to implement some of the changes recommended by the 2011 Bahrain Independent Commission of Inquiry, according to Sir Nigel Rodley, a human-rights lawyer who took part in the commission.


One detainee, Talib Ali Mohammed, 37 years old, was arrested in November on suspicion of involvement in coordinated bombings in Manama that month that killed two expatriate workers.

Over 16 days of interrogation in the Central Intelligence Department building in the Adliya district of Manama, Mr. Talib was beaten repeatedly and tortured, according to his wife, Fatima Ebrahim, and his lawyer, Sayed Hashin Saleh, who have seen Mr. Talib in prison and spoken with him by phone. Mr. Talib eventually confessed to charges including possessing explosive material and forming a group with the intention of harming others.


Ahmed Abdullah, a 24-year-old gymnasium worker, was arrested in November and accused by authorities of involvement in the bombings. According to his brother Ibrahim, who has visited him in prison and spoken to him by phone, Mr. Abdullah was blindfolded for nearly 20 days in the CID building in Adliya, where he was beaten repeatedly, and forced to stand for long periods until he signed a confession.

There is now new leadership at the Department of State. Will we see a stronger condemnation of torture by the Bahrain government and support for Rodley’s call for a new commission of inquiry over the new torture accusations, or will we get the same weak platitudes we saw from Foggy Bottom last year?

Bahrain continues to profess its innocence. In one of the most craven, idiotic defenses by a government ever, the Journal carried this denial:

Minister of State for Information Affairs Samira Ibrahim Bin Rajab dismissed the allegations. “This is not our culture, not our attitude or our behavior,” she said. “We are very civilized, educated people.”

Civilized, educated people never torture. They rely on enhance interrogation techniques that are perfectly legal. Just ask John Yoo. He’ll confirm that in an instant and have a follow-up memo for you tomorrow that retroactively authorizes any actions you need approved.

Is Democracy the Problem, or Money-Corrupted Governance?

I’ve been pondering this NYT story–which is presented as news yet which in fact is analysis attempting to provide a general explanation for protests in democracies–since it came out. Its general explanation for why so many people are protesting is that people–primarily youth–have grown disillusioned with voting.

Hundreds of thousands of disillusioned Indians cheer a rural activist on a hunger strike. Israel reels before the largest street demonstrations in its history. Enraged young people in Spain and Greece take over public squares across their countries.

Their complaints range from corruption to lack of affordable housing and joblessness, common grievances the world over. But from South Asia to the heartland of Europe and now even to Wall Street, these protesters share something else: wariness, even contempt, toward traditional politicians and the democratic political process they preside over.

They are taking to the streets, in part, because they have little faith in the ballot box.

Note, the title of the article (which presumably the authors didn’t write) refers to a “scorn for vote,” but even this last sentence focuses on the ballot box, rather than the system the ballot box supports. The article doesn’t offer any polling to show this generation (or even just protest participants) are objecting to voting, per se, nor does it question why the record number of youth who came out to vote in the US in 2008 are now among those occupying Wall Street. Rather, it offers these quotes from a protest participants.

“Our parents are grateful because they’re voting,” said Marta Solanas, 27, referring to older Spaniards’ decades spent under the Franco dictatorship. “We’re the first generation to say that voting is worthless.”


“We elect the people’s representatives so they can solve our problems,” said Sarita Singh, 25, among the thousands who gathered each day at Ramlila Maidan, where monsoon rains turned the grounds to mud but protesters waved Indian flags and sang patriotic songs.

“But that is not actually happening. Corruption is ruling our country.”


Mr. Levi, born on Degania, Israel’s first kibbutz, said the protests were not acts of anger but of reclamation, of a society hijacked by a class known in Hebrew as “hon veshilton,” meaning a nexus of money and politics. The rise of market forces produced a sense of public disengagement, he said, a feeling that the job of a citizen was limited to occasional trips to the polling places to vote.

“The political system has abandoned its citizens,” Mr. Levi said. “We have lost a sense of responsibility for one another.”

All three of these speakers are talking about something more than democracy. They’re talking about democracy that has been delegitimized by its insulation from voters; two specify that corruption is the culprit.

In other words, the article claims to report something about protestors’ attitude towards democracy, while mostly downplaying the role that money has had in the failed governance that results from that democracy, though the protests focus on the latter.

The authors fail to distinguish between democracy and capitalism in other ways, too. In one case, for example, they use a quote talking about capitalism to support a claim they make about voting.

Frustrated voters are not agitating for a dictator to take over. But they say they do not know where to turn at a time when political choices of the cold war era seem hollow. “Even when capitalism fell into its worst crisis since the 1920s there was no viable alternative vision,” said the British left-wing author Owen Jones. [my emphasis]

And while they say, “the protest movements in democracies are not altogether unlike those that have rocked authoritarian governments this year,” they only examine the technological similarities, the reliance on social media in both. They don’t bother to consider the commonality between Tunisians demanding jobs, Israelis demanding affordable housing, Europeans fighting austerity or (in the case of London’s riots) for some kind of future. And while they link to news on Occupy Wall Street, they don’t even mention Wisconsin, perhaps because the involvement of unions and middle class teachers would spoil their desired narrative, which claims protestors are also bypassing unions.

A globalized economy has presented similar problems leading to similar protests in democracies and authoritarian regimes alike, but the NYT’s reporters want to claim this is about democracy and not economics.

All of which builds to their judgment, one terribly sourced paragraph spinning these protests as a profoundly undemocratic movement.

While the Spanish and Israeli demonstrations were peaceful, critics have raised concerns over the urge to bypass representative institutions. In India, Mr. Hazare’s crusade to “fast unto death” unless Parliament enacted his anticorruption law struck some supporters as self-sacrifice. Many opponents viewed his tactics as undemocratic blackmail. [my emphasis]

“Critics have raised,” “many opponents viewed.” None of them named or quoted in the article, but all critically deployed to interpret the evidence the reporters set forth as being primarily about democracy and not about so-called capitalism (otherwise known as elite looting).

For the record, I do believe there’s commonality among these protests. Not just the ones the authors puzzle through in Israel, India, and Europe, but also those in Madison, Wall Street, Egypt, and Tunisia. I do believe it’s worth reflecting on this commonality. But I find it telling that an article published in the most elite news institution and complaining that, “protesters have created their own political space online that is chilly, sometimes openly hostile, toward traditional institutions of the elite,” interprets the commonality here as a rejection of democracy, not a rejection of elite looting.

Our Counterterrorism Policies Will Make Impact of Climate Change Worse

What place does this sound like?

Ruling elites … do not see climate change as an immediate threat to their authority. They therefore feel free to take an opportunistic attitude toward climate change, supporting climate change mitigation policies that have collateral economic or political benefits to their particular interests.

Though it could be, it is not an indictment of our own country’s refusal to do anything about climate change. Rather, it’s one of a series of climate change studies and conferences the National Intelligence Council contracted to have done. This one describes the self-serving actions of the pre-Arab Spring authoritarian elite of North Africa.

As Steven Aftergood reported, the CIA is hiding the climate change analysis they’re doing. They just rejected a FOIA for their climate change reports based on a claim that everything they have done is classified. So these reports, prominently labeled, “This paper does not represent US Government views,” are one of the only public reads about what the intelligence community is doing with climate change.

Those contractor studies are interesting for several reasons. First, check out how they define their regions:

  • China
  • India
  • Russia
  • Southeast Asia and Pacific Islands
  • North Africa
  • Mexico, the Caribbean, and Central America

The impact of climate change on the US, Europe, much of the Middle East, and most of Africa are all missing (or, at least, not public).

Shouldn’t someone (not the CIA, which can’t, but perhaps DOE) start thinking about how climate change will affect security in the US? How do you rationalize not including the Middle East (where water is already is source of conflict between Israel and its neighbors) or the Horn of Africa (where climate-related issues discussed in the North Africa studies have presented predictably catastrophic problems in countries that already pose other national security challenges to the US)? Why study India rather than South Asia as a whole, particularly given that Bangladesh will be one of the most impacted countries and (as reflected in the India report) will present India with a serious refugee problem. In short, there are real, critical gaps in the way the intelligence community at least publicly thinks of the potential impact of climate change.

I checked out the North Africa reports (commissioned report, conference report) to see how the intelligence community viewed the region two years before the Arab Spring. True, these reports analyze the impact of things like drought on agriculture and the impact of that on stability, but such analysis largely parallels the impact of neoliberal economic policies on agriculture and therefore on stability. Here’s what the NIC was hearing about climate change and Ag and stability two years before the Arab Spring (these quotes come from the conference report):

An acute state failure to address climate change that results in intolerable conditions for significant segments of the population may constitute a sociopolitical tipping point, in essence a breaking of the social compact between North African states and civil society. At that point, civil actors may determine that fundamental systemic change is necessary. The results of such a situation will depend on the specific reactions by state elites and by the public; reform, repression, or revolution are all possibilities. A combination of climatic stress and inadequate state responses over the next two decades could prove the catalyst for a major sociopolitical shift in North Africa. On the other hand, North Africans tend to hold a religiously based view that “what will be, will be.” Owing to this fatalistic mindset, North Africans are unlikely to blame the state for climate related stresses, making it more difficult to attain the aforementioned tipping point.

Much later, the report predicts that the ancillary effects of climate change will be the cause of social stress.

The implications of climate change in North Africa—notably migration, stress on both rural and urban areas, unemployment, and increased resource competition—are likely to generate volatile sociopolitical conditions that will pose significant threats to the existing political structure. The responses of North African states to these threats may be more decisive for the fate of the region than their direct responses to climate change impacts. North African states have robust capacity to maintain social control in the face of domestic challenges and destabilization. Regimes depend on a combination of entrenched patronage systems, robust mukhabarat (security) apparatuses, and the support of external allies—a combination that has proven highly effective at maintaining political control. They have a track record of effectively suppressing dissent and unrest or remaining resilient where unrest has persisted, such as the civil conflict in Algeria.

States in the region may seek to suppress or distort information on climate change-related challenges. They seek to control access to any information that could provide a basis for opposition to the state, even information as seemingly innocuous as census data. The proliferation of new media and alternative information sources, however, will make it difficult to maintain such censorship. [my emphasis]

Particularly given our own IC’s failure to take the warnings of unrest expressed via social media social media seriously, I find the warning that North African regimes would find it hard to censor this social unrest prescient.

And I find it richly ironic that the IC notes other countries would “seek to suppress or distort information on climate change-related challenges” when the CIA is doing just that in the US.

But I also find the description of these regimes’ reliance on their allies chilling. This report always describes these regimes, several of them key allies of ours, as badly repressive regimes.

Although the level of repression varies between states, with Tunisia and Libya the most extreme, and has varied cyclically over time, authoritarian regimes are well entrenched in every state in the region.

The conference report acknowledges that the US focus on terrorism has narrowed its diplomatic focus with these countries, which in turn has strengthened the security apparatuses in the region–precisely the source of the repressive strength of the countries.

Security issues are the primary focus of US relations with North African states. The predominance of security and military concerns has led to disproportionate US engagement with security apparatuses in the region, strengthening regimes in ways that may damage long-term prospects to meet the challenges of climate change. US policy in the region has become even more security-centric as a result of the continuing struggle against radical Islamic terrorism. While terrorism has deepened US security ties with states in the region, it has also narrowed the scope of US engagement, which may not be in the long-term interests of either party.

And then the report incorrectly suggests that the only likely challenge to these regimes if they fail to respond adequately to climate change would be Islamists.

Islamist groups have emerged as the only viable opposition force because they have resisted state cooptation and because the state has blocked other avenues for social mobilization. In addition, they have established a track record of effective humanitarian responses to mudslides, earthquakes, and other natural disasters, often providing immediate medical, shelter, and food aid that are normally the responsibilities of the state. In many cases Islamist groups may fill the void left by inadequate state responses or the weakness of other types of potential civil responders. Moderate Islamist groups could play a constructive role, providing highly visible humanitarian assistance that empowers autonomous civil actors and contrasts with ineffectual state responses, thus pressuring state actors to respond more effectively. Moderate Islamists could use the climate change mitigation issue to bolster their argument that existing North African governments are illegitimate and exploitative, creating momentum for political reform.

On the other hand, Islamic extremists across the region may exploit climate change’s destabilizing impacts and ineffective state responses to promote the spread of militancy and anti-regime violence. Indeed, Islamist militants could point to climate-induced catastrophes as evidence of God’s wrath against “apostate regimes” whose un-Islamic behavior has plunged the region into desperate circumstances.

In other words, while the report doesn’t lay out the the logical case it makes explicitly, it nevertheless argues that:

  • The repressive nature of these regimes may make them less likely to respond adequately to climate change
  • Our single-minded focus on terrorism tends to make these countries even more repressive
  • If these countries don’t respond to climate change, it may provide an opportunity for precisely the Islamists our single-minded counter-terrorism focus is designed to combat

In other words, this conference report suggests (though does not say so explicitly, perhaps because it was written by contractors intent on getting paid) that in the presence of a stress like climate change, our counter-terrorism approach may be self-defeating.

Now, again, this report wasn’t written by our spooks and it “does not represent US Government views.” Our policy makers may not agree with this report’s analysis, or they may be ignoring it (seeing no “collateral political or economic benefits to their particular interests”). And if you buy my premise that the stress of climate change is similar to the stress caused by an embrace of neoliberalism, then the report badly underestimated both the success of those challenging these regimes and the centrality of Islamists in these countries.

There’s a lot else that could be said about these reports (such as their too-narrow focus on the Ag in each particular country, when recent food price shocks make it clear such stress will play out at broader levels).

But more generally, the report suggests that our counterterrorism policies are making countries around the world less resilient to climate change (and so presumably to a range of other stresses as well).

The World Should Have Revolted Over America’s Illegal War on Iraq

You may have seen discussions about this project around the Toobz. In it, scholars use supercomputers to analyze the tone of news coverage. Their results from Egypt and Tunisia–showing low sentiment right before this year’s revolutions–suggest you can predict volatile events with such analysis.

I decided to look further at the study, not least, because of Dianne Feinstein’s complaint earlier this year that the CIA had totally missed stirrings of rebellion in both countries.

Feinstein set a skeptical tone at the opening of the hearing, saying Obama and other policymakers deserved timely intelligence on major world events. Referring to Egypt, she said, “I have doubts whether the intelligence community lived up to its obligations in this area.”

After the hearing, Feinstein said she was particularly concerned that the CIA and other agencies had ignored open-source intelligence on the protests, a reference to posts on Facebook and other publicly accessible Web sites used by organizers of the protests against the Mubarak government.

Speaking more broadly about intelligence on turmoil in the Middle East, Feinstein said, “I’ve looked at some intelligence in this area.” She described it as “lacking . . . on collection.”

According to DiFi, the CIA missed the Arab Spring because they weren’t monitoring open source materials (an argument that WikiLeaks cables seem to confirm). And this study is all the more damning for our intelligence community, because this study uses their own (actually, Britiain’s) open source collection.

Recognizing the need for on–the–ground insights into the reaction of local media around the world in the leadup to World War II, the U.S. and British intelligence communities formed the Foreign Broadcast Information Service (FBIS — now the Open Source Center) and Summary of World Broadcasts (SWB) global news monitoring services, respectively. Tasked with monitoring how media coverage “varied between countries, as well as from one show to another within the same country … the way in which specific incidents were reported … [and] attitudes toward various countries,” (Princeton University Library, 1998) the services transcribe and translate a sample of all news globally each day. The services work together to capture the “full text and summaries of newspaper articles, conference proceedings, television and radio broadcasts, periodicals, and non–classified technical reports” in their native languages in over 130 countries (World News Connection, 2009) and were responsible for more than 80 percent of actionable intelligence about the Soviet Union during the Cold War (Studeman, 1993). In fact, news monitoring, or “open source intelligence,” now forms such a critical component of the intelligence apparatus that a 2001 Washington Post article noted “so much of what the CIA learns is collected from newspaper clippings that the director of the agency ought to be called the Pastemaster General.” (Pruden, 2001)

While products of the intelligence community, FBIS and SWB are largely strategic resources, maintaining even monitoring coverage across the world, rather than responding to hotspots of interest to the U.S. or U.K. (Leetaru, 2010). A unique iterative translation process emphasizes preserving the minute nuances of vernacular content, capturing the subtleties of domestic reaction. More than 32,000 sources are listed as monitored, but the actual number is likely far lower, as the editors draw a distinction between different editions of the same source. Today, both services are available to the general public, but FBIS is only available in digital form back to 1993, while SWB extends back more than three decades to 1979, and so is the focus of this study. During the January 1979 to July 2010 sample used in this study, SWB contained 3.9 million articles. The only country not covered by SWB is the United States, due to legal restrictions of its partner, the CIA, on monitoring domestic press.

If you believe that study author Kalev Leetaru’s research is valid (I think it’s very preliminary), then you basically grant that using his data analysis methods would have warned our intelligence services that the unrest in North Africa was exceptionally high.

But that’s not what I found most intriguing about Leetaru’s research.

To measure whether the data from the SWB was an outlier, Leetaru compared how trends he saw from that data compared to the NYT and English language news more generally.

And as he explains, they generally track during this period, though with key deviations.

To verify that these results are not merely artifacts of the SWB data collection process, Figure 4 shows the average tone by month of Summary of World Broadcasts Egyptian coverage plotted against the coverage of the New York Times (16,106 Egyptian articles) and the English–language Web–only news (1,598,056 Egyptian articles) comparison datasets. SWB has a Pearson correlation of r=0.48 (n=63) with the Web news and r=0.29 (n=63) with the New York Times, suggesting a statistically significant relationship between the three. All three show the same general pattern of tone towards Egypt, but SWB tone leads Web tone by one month in several regions of the graph, which in turn leads Times tone. All three show a sharp shift towards negativity 1–24 January 2011, but the Times, in keeping with its reputation as the Grey Lady of journalism, shows a more muted response.

That is, for these events, local coverage was both more attuned to a change in sentiment and more reflective of the volatility of it. Or to put it another way, the NYT was slow to consider Egypt a major story, and never thought it was as big of a deal as the rest of the world did.

A far more interesting comparison of how the NYT outlook compared with the rest of the world comes in these two graphs, which show the NYT sentiment from 1945 to 2005 and the SWB sentiment from 1979 to 2010 (caution–neither the X nor Y axes here use the same scale; click to enlarge or go to the study for larger images).

You can sort of pick out events that might be driving sentiment on both scales. And they don’t entirely line up. Just as an example, the US seems to have reacted far more strongly–2 deviations as compared to .5 deviation–to what appears to be the first Gulf War in January 1991.

But note where both data sets converge more closely: with our second war against Iraq, with even the chief cheerleader for war, the NYT, measuring in the high 2 deviations from the mean in early 2003, and the international SWB measuring almost 2 deviations from the mean.

Significantly, the study shows that Egyptian sentiment before they revolted was in the 3+ range–more incensed than we were with the Iraq invasion, but not by much; whereas sentiment in Tunisia and Libya was less negative. Were we that close to revolting?

Now, I could be misreading both the stats and the explanation for the global bad mood as we lurched toward war against Iraq (though it also shows up in the Egyptian and Tunisian graphs; the sentiment is least severe in Libya). But if I’m not, it raises questions about what was driving the sentiment. In Europe especially and even in the US, there were huge protests against the war, though we never seemed all that close to overthrowing the war-mongers in power. I wonder, too, whether the sentiment also reflects the ginned up hatred toward Saddam Hussein. That is, it may be measuring negative sentiment, but partly negative sentiment directed against an artificial enemy.

So are these graphs showing that we were even closer to revolt than those of us opposed to the war believed? Or is the NYT graph showing that warmongers reflect the same nasty mood as people attempting to prevent an illegal war?

In any case, the NYT coverage reflected the crankiest mood in the US of the entire previous half century, significantly worse than the VIetnam period. I knew I was cranky; I wasn’t entirely sure everyone else was, too.

The Global Crisis of SOME Institutional Legitimacy

Felix Salmon has a worthwhile (but, IMO, partly mistaken) post on what he deems “the global crisis of institutional legitimacy.” I think he’s right to see this as a significant challenge to our current political economy.

While watching another Arab government get toppled on Sunday evening — this time that of Muammar Gaddafi, in Libya — I was also reading George Magnus’s excellent note for UBS, entitled “The Convulsions of Political Economy”; you can find it chez Zero Hedge.

Convulsions is right — not only in the Arab world, of course, but also in Europe and the US. And the result is arguably the most uncertain outlook, in terms of the global political economy, since World War II ended and the era of the welfare state began.

As Magnus says:

It seems that we are having sometimes esoteric tiffs between Keynesians and Austrians about if and how governments should sustain jobs and growth. But, deep down, we are having a much more significant debate as we are being forced to redefine what we think about the rights and obligations of citizens and the State.

Most fundamentally, what I’m seeing as I look around the world is a massive decrease of trust in the institutions of government.

But I think Salmon makes two mistakes. First, he maintains an unwarranted distinction between the Arab Spring and the UK riots.

Where those institutions are oppressive and totalitarian, the ability of popular uprisings to bring them down is a joyous and welcome sight. But on the other side of the coin, when I look at rioters in England, I see a huge middle finger being waved at basic norms of lawfulness and civilized society, and an enthusiastic embrace of “going on the rob” as some kind of hugely enjoyable participation sport. The glue holding society together is dissolving, whether it’s made of fear or whether it’s made of enlightened self-interest.

From the perspective of the underclass in our society, it has been some time since “enlightened self-interest” counseled compliance. And from most perspectives, it’s clear that the elites, not the underclass, were the first to wave a huge middle finger at basic norms of lawfulness.

A more problematic error, though, is Salmon’s claim that corporations have retained their legitimacy.

Looked at against this backdrop, the recent volatility in the stock market, not to mention the downgrade of the US from triple-A status, makes perfect sense. Global corporations are actually weirdly absent from the list of institutions in which the public has lost its trust, but the way in which they’ve quietly grown their earnings back above pre-crisis levels has definitely not been ratified by broad-based economic recovery, and therefore feels rather unsustainable.

As a recent Pew poll shows, Americans are just as disgusted with banks and other large corporations as they are with their government.

While anti-government sentiment has its own ideological and partisan basis, the public also expresses discontent with many of the country’s other major institutions. Just 25% say the federal government has a positive effect on the way things are going in the country and about as many (24%) say the same about Congress. Yet the ratings are just as low for the impact of large corporations (25% positive) and banks and other financial institutions (22%). And the marks are only slightly more positive for the national news media (31%) labor unions (32%) and the entertainment industry (33%).

Notably, those who say they are frustrated or angry with the federal government are highly critical of a number of other institutions as well. For example, fewer than one-in-five of those who say they are frustrated (18%) or angry (16%) with the federal government say that banks and other financial institutions have a positive effect on the way things are going in the country.

But there are institutions that Americans still trust: colleges, churches, small businesses, and tech companies.

Distinguishing between those institutions (government and big corporations) people distrust and those (churches, small businesses, and tech companies) they do is important for several reasons. First, because it prevents us from assuming (as big corporations might like us to) that Americans will be content with corporatist solutions. People may or may not like the the post office, but there’s no reason to believe they like FedEx, Comcast, AT&T, or Verizon any more, particularly the latter three, which all score very badly in customer satisfaction. (Update: as joberly points out, Pew found that the postal service was by one measure the most popular government agency, with 83% of respondents saying they had a favorable view of the postal service.)

Such polling also suggests where Americans might turn during this convulsion. Barring Apple buying out the federal government, it seems likely Americans, at least, will turn to local institutions: to their church, their neighborhood, their local businesses.

That’s got some inherent dangers–particularly if people decide they want to change my governance with their church. But it also provides a nugget of possible stability amid the convulsion, one that might have salutary benefits for our environment and economy.

Apple aside, it’s the big institutions that have lost their institutional legitimacy. But we’re not entirely without institutions with which to rebuild.