Shiny, Shiny Data: The Thrill of the Chase

Why Washington and Silicon Valley must work together to truly understand the world.

One of the most striking revelations of the Edward Snowden disclosures has been the single-minded focus of the U.S. intelligence community on collection: on hoovering up all global communications, but with the concept of analysis -- of what to actually do with all those communications -- relegated to an afterthought.

Gleeful bragging and schoolboy taunts of hacking coups permeate the disclosed documents, but discussion of how the captured data can actually be used is far more mundane and subdued. The thrill clearly lies in the chase of new data, rather than in the actual analysis of the material obtained. In fact, an image emerges of a community struggling merely to read its highest-priority intercepts, let alone assess their contents.

An exchange among analysts monitoring the Lashkar-e-Taiba terrorist group notes, "Most of [the intercepted communications traffic] is in Arabic or Farsi, so I can't make much of it." In an era when web browsers instantly translate over 80 languages, it seems absurd that the analysts tracking one of the largest terrorist organizations in South Asia can't even read the language. The prioritization of collection has led us to a point where the United States can acquire a terrorist group's internal communications at will, but this becomes for naught when the analysts don't know what those communications say.

While the National Security Agency's surveillance techniques may awe in their technological sophistication, they are in reality nothing more than a modernization of age-old collection tradecraft for the computer era. Hacking into computers and downloading files is arguably no different from breaking into an office and photocopying papers. Maintaining lists of software vulnerabilities is no different from curating bypasses for alarm systems and safes. And intercepting computers in transit to install keystroke tracers is little different from hijacking furniture shipments to install listening devices.

Even the most controversial of the National Security Agency (NSA) programs, the bulk collection of phone metadata, traces its roots back nearly 70 years to an earlier NSA effort, Project SHAMROCK, which monitored all international telegraphs crossing the borders of the United States (which in turn was based on the post-World War I "Black Chamber"). Lasting nearly 30 years, SHAMROCK collected over 150,000 telegrams per month at its peak, and public discovery of the operation led to the creation of the FISA court system that is now central to the modern surveillance apparatus. Even the frenzied hyperbole was the same, with Sen. Frank Church, chairman of the Senate Committee to Study Governmental Operations With Respect to Intelligence Activities, condemning the program as "probably the largest government interception program affecting Americans ever undertaken."

The problem is that this modernization of collection tradecraft has not been matched with a modernization of analysis tradecraft. The intelligence community has become so enamored with its collection gadgets and technology that it has forgotten that the true purpose of intelligence is not merely to collect, but to actually create intelligence from all that collected material, which requires interpretation and analysis. In essence, with a single-minded focus on chasing new information, James Bond has become Melvil Dewey, simply collecting and archiving the world's information. This has not gone unnoticed, as Josh Kerbel, chief analytic methodologist for the U.S. Defense Intelligence Agency, has written extensively on the community's "inability to think big" and its need to "better balance its traditional focus on entities with a greater appreciation for phenomena … [to] recognize the futility in an excessively insular and closed organization trying to understand and make sense of an open, complex world."

The world is a far more complex place than it was half a century ago at the founding of the American intelligence apparatus, when it was just the Allies versus a handful of villainous nation-states. The Hollywood image of a lone spy in a telephone exchange closet with a pair of headphones and a notepad jotting down Vladimir Putin's secret plans for eastern Ukraine and relaying them back to Washington simply isn't possible in a world where there are more targets to monitor than spies to watch them. In today's data-drenched world, it is increasingly subnational actors with fluid geographies that threaten America's interests, and Twitter alone publishes more words in a single day than were printed in the entire New York Times over the past half-century. A private individual can post a video to YouTube that wracks an entire region and unleashes a torrent of anti-American sentiment faster than analysts even realize there is a problem. In short, today's world moves too fast and involves too many actors for humans sitting at their desks to actually understand the flood of information that our modernized data plumbing is delivering to our doorstep.

Why is it that collection tradecraft has modernized to keep pace with technological innovation, while analysis has advanced little from the human-driven qualitative methodology of half a century ago? One likely answer is that when it comes to collection, the intelligence community has stepped back and allowed technologists to lead the way, while analysis has been kept firmly within the death grip of those who view computers as a threat to their job security. Even as a new generation of data-savvy analysts increases demand for technological innovation, management often views such tools with suspicion, afraid they will displace and shrink their human-centric empires, rather than seeing the potential of the tools to refocus analysts on actual analysis.

The increasing priority given to information collection has generated a need for translation across disciplines -- to bring together the technical, analytic, and policymaking communities to work together as a team, rather than as separate enterprises. Silicon Valley produces a wealth of new tools each day to store, manage, visualize, and analyze the big-data universe, but it approaches the data revolution purely through the lens of technological innovation. Analysts too often view data tools as a threat to their jobs, fixate on those tools' limitations, don't possess the technical skills to use them, or fall victim to marketing hype, using tools and data blindly without understanding how to properly apply them. Policymakers just want answers, but often ask questions that are not aligned well with today's technological capabilities and view data's inability to answer their questions as proof of its immaturity.

What we need is to increase Washington's "data literacy" and Silicon Valley's "application literacy" to bring the two worlds together in the pursuit of data-driven intelligence and policymaking.


I sat through a presentation last year featuring a sequence of visually stunning maps and charts depicting changing Syrian rebel views toward the Syrian regime. Yet, not a single visualization contained a mention of source data, analysis algorithms, or the confidence interval of the data other than a mention that they were based on "billions of observations."

When I raised my hand and asked how street-level data on rebel views had been assessed using open sources, the response was that the Twitter fire hose (containing "billions" of tweets) had been scanned for English-language tweets originating in Syria, resulting in a few hundred tweets, and that the software used to codify the tweets for mapping specifically warned that it would generate invalid results on social media data. When I noted that I thought the rebels were using Arabic-language Facebook posts, the answer was that Twitter data was easier to access and that no one on the team spoke Arabic and that they selected the software because it was easy to use. Thus, what started as a sequence of alluring visualizations touting street-level results and based on "billions of observations" became a handful of English-language tweets in a country where few speak English and Facebook is the primary communications platform of the rebels.

The researchers behind these maps did not mean to mislead. They simply suffer from "shiny new object" syndrome -- well-meaning technologists grab data sets and off-the-shelf tools and create technological breakthroughs that often bear no resemblance to ground reality. Had this been a traditional intelligence assessment, it would have come with a detailed assessment of the underlying sources, assumptions, analyses, and possible caveats of the results, and much of the conversation would have focused on how likely the data reflected the overall rebel population. Instead, the lack of "application literacy" among technologists means that data-driven analyses are often based on whatever data and tools are easiest to obtain and use. The concept of how closely those data sets or algorithms reflect reality never comes into play, as the computer science world centers on software rather than solutions.

If intelligence analysts and policymakers understood more about data, they would be able to see past the shiny animated maps and ask the hard questions about what lies underneath. In doing so they might become more confident in the results and more willing to allow data-driven analyses to play devil's advocate. A number of Wall Street forecasts correctly predicted the 2008 financial collapse, but were dismissed by human analysts who trusted their own gut feelings over data. Similarly, forecasts of the 2011 Egyptian revolution were crippled by subject matter experts who argued that Mubarak could never fall, leading to computer forecasting models being built only to forecast what would happen if he stayed in power. The megastore Target deployed highly sophisticated computer-intrusion software that correctly detected its credit card breach, but whose alerts were dismissed when they conflicted with human intuition.

Nearly half a century of investment in data-driven conflict forecasting has yet to lead to a system that can forecast the Arab Spring or the Russian annexing of Crimea. Instead, the political science community touts computer models with 80 percent accuracy because they correctly predict each day that Syria will have conflict the following day, or announce they correctly predicted Venezuela's unrest after two weeks of New York Times headlines chronicling the spiraling violence. If one predicts that no country will undergo a coup at any point during the coming year, that's technically 196 countries times 365 days = 71,540 predictions. If even 10 countries go under this year, that's still 99.986 percent accuracy. To a data-driven political scientist 80 percent accuracy may be viewed as a major technical achievement, but to a policymaker, being told Venezuela may have problems two weeks after it has already unraveled is not predictive and it undermines their trust in what data is capable of. Increasing application literacy would help the quantitative political science community begin to bridge the gap between technical achievement and useful results, while increasing data literacy would help policymakers better communicate to them what constitutes a useful and actionable result.

In short, we need to do a better job of getting Silicon Valley to better understand the applications of its technology, while we need to do better at helping policymakers understand what actually goes into the pretty pictures they get back from all of these tools. Perhaps the intelligence community should take a page from the playbook of its favorite hero, James Bond. After 50 years of devolving from intellect and action to a two-hour technology show of gadgets, the Bond franchise was rebooted. To the modern Bond, gadgets are simply tools to help him achieve his needs, with the focus on analysis, assessment, and action. To stay relevant in an ever-evolving and complex world, the U.S. intelligence community must bring Silicon Valley and Washington together to help it "reboot" itself to refocus on understanding the world, and in doing so, become less like Melvil Dewey and more like James Bond again.

Patrick Lux/Getty Images


Washington's Sleeping Sickness

The greatest force at work in America's capital is inertia.

In the seat of American power, power spends most of its time on its seat. That's because, as Henry Kissinger once observed, the greatest force at work in America's capital is inertia.

It handily trumps partisanship and also leaves the more positive drivers of action that one might hope for in a government -- such as leadership, creativity, or moral courage -- coughing and wheezing in the dust.

One clear, compelling illustration of this is on display in this issue of Foreign Policy. Shawn Brimley and Paul Scharre's thoughtful and thought-provoking visual feature, "Ctrl+Alt+Delete: Resetting the U.S. Military," asks the sensible question: What would the military look like if we were to design it today from scratch? Certainly, the piece observes, the military would not include the breathtaking redundancies, efficiency-killing bureaucracy, and obsolete systems of today's bloated defense apparatus. You might disagree with the feature's conclusions about what a right-sized, technologically up-to-date, doctrinally sound military, conceived and prepared to ensure America's safety and worldwide interests, might look like. That's perfectly reasonable. But it is impossible not to conclude that the feature's call to debate what the military should look like and then implement the agreed-upon changes makes sense.

It makes economic sense because the United States spends more on defense than all other major powers combined -- even though very nearly all of them are our allies. It makes national security sense because the threats the country faces are changing and because emerging powers are not the slaves to legacy systems that the United States is. It makes political sense because such a reform process is the very essence of good governance and would free up resources for endeavors that could help broader cross-sections of the American population.

Yet the one thing that we know about this idea is that it's never going to happen.

That is because it would require the kind of far-reaching change that the government is terrible at achieving. It would involve confronting moneyed, entrenched interests in the private sector as well as the Pentagon, which kills ideas that threaten its core programs more efficiently than it does any foreign enemy. This is the military-industrial complex that Dwight Eisenhower warned us about as he prepared to leave office -- except today it is bigger and more powerful than it has ever been.

This does not mean, of course, that there are no hugely creative military leaders who have contemplated just the kind of changes that are needed. In Washington, however, strength lies with the opponents rather than the proponents of change. And the opponents possess the ultimate political weapon of mass destruction: They can accuse leaders who want to challenge the status quo of making America weak. So we're left with all the systems the country has accumulated to counter every post-World War II defense challenge: multiple air forces, multiple expeditionary forces, multiple cyber commands.

We can't afford what we have. We don't need much of what we have. The country is misallocating resources in precisely the same way as other nations that have historically made themselves vulnerable, both militarily and economically. Yet any positive, proactive change will exist only on the margins until some crisis demands otherwise, while other supposed "innovations" will in fact just maintain things as they are and have always been. All you need to do is flip back in the magazine and spend a few minutes reading Rosa Brooks's excellent and eye-opening exploration of the Army's regionally aligned forces (RAF) strategy -- convoluted, ill-defined, and dubious -- to see the creative genius of Washington in action. As Brooks concludes, the strategy may be of more value in protecting the Army from budget cuts than in protecting America from future enemies.

Washington's debilitating strain of sleeping sickness affects other parts of the government too.

Indeed, the political system hates action more than nature abhors a vacuum. Both political parties would no doubt agree that the country cannot afford the retirement health-care system it has, even if they have different answers for how to fix the problem. Both parties also have to agree that it is in America's interest to invest in highways and bridges in a meaningful way, which hasn't been done since the Eisenhower era. And both must acknowledge that having half the minority students in inner cities not graduating from high school is a formula for social catastrophe. Yet all these issues remain largely unaddressed.

Even outside government, we have built massive apparatuses designed to assert that some issues are not issues at all, in order to protect the interests of a few. Climate change is one such issue: enormous and indisputable, yet disregarded thanks to political pressure from a well-funded alliance with an affinity for profit and an allergy to science. Inequality rivaling that of the Gilded Age is another issue, as is the country's inability to create jobs for the middle class, because the rich use their influence to prioritize their interests above those of the rest. The U.S. financial system, meanwhile, is corrupt to its core, but its leaders act with impunity -- sometimes with the assistance of members of the government.

With each of these issues, the common sense that keeps a child from touching an open flame or playing in traffic would suggest that actions are urgently needed. But they are not and will not be forthcoming. In no small part, this is because we have given a handful of people disproportionate influence over elections. The Supreme Court's recent McCutcheon decision, like Citizens United before it, enshrines the idea that money is speech -- therefore those with more money now have more say in the country's national affairs. We have also institutionalized the right-left, no-compromise polarization of the political system via gerrymandering and Capitol Hill customs. This has undercut reasoned debate and votes that could produce progress.

But our inertia is caused by something else too. Call it superpower smugness. Or just call it complacency. We don't create, or even really demand, genuine change in America because things seem to be working well enough. America is the world's richest and most powerful nation. When we have crises, we recover from them. The country can make catastrophic errors of judgment in places like Iraq and Afghanistan and absorb the costs and collateral damage to its national image. Furthermore, the private sector, for all its self-interested political behavior, is infused with genius and a stunning capacity to reinvent itself. It has compensated time and again for Washington's sloth and willful ignorance.

So we have been numbed into believing that black is white, that a failing system is functioning, that we, alone among nations, are invulnerable to our own stupidity. If there were ever a flaw in a strategy for national security, this is surely the most pernicious. The only antidote is a healthy dose of courage to act first with good sense and second with a spirit for reform, not just within the military but among average civilians as well.

Illustration by Matt Chase