Playing Nostradamus

Big data show that history does indeed repeat itself. What does that mean for foreign policymaking -- and tackling crises from Ukraine to Syria?

Working in policymaking is a lot like being a real-life time traveler: It is an inherently forward-looking process in which decisions are made today based on an estimate of what tomorrow might look like. Yet as the finger-pointing that accompanies every global crisis, from Ukraine to Syria, makes clear, humanity's ability to forecast the future is poor at best.

This doesn't have to be the case. From the ancient Egyptian symbol of the Ouroboros (a serpent eating its own tail) to the Hindu concept of the Yuga (cyclical epochs) to the science-fiction world of Isaac Asimov's "psychohistory," thinkers have long reasoned that history repeats itself and can thus be measured and predicted. In the case of psychohistory, it was posited that, much like weather forecasts, the future actions of human society could be estimated at high accuracy if there was just sufficient data and computing power. 

So has the failure to accurately forecast what will happen in the world -- and to make policies to prevent catastrophe -- merely been due to a lack of information about historical cycles? The answer, it seems, is "yes."

The Global Database of Events, Language, and Tone (GDELT) project, which I founded, monitors the world's broadcast, print, and web news from nearly every country in more than 100 languages and compiles a daily list of more than 300 categories of events -- from riots to appeals for peace -- down to the city level. It currently has more than a quarter-billion records from 1979 to the present, all of them available in Google's BigQuery analysis platform. The resolution, time scale, and geographic coverage of this database makes it ideal for exploring whether history really does fall into regular patterns, and if those patterns are sufficiently robust to forecast the future.

Turn back the clock to Jan. 27, 2011: Egypt is wracked with anti-government protests, and analysts are furiously trying to determine whether there will be a quick resolution or whether the protests will bring months of turmoil to the country. Using GDELT and BigQuery to answer this question, a timeline could be created measuring the intensity of unrest (defined as material, nonverbal conflict) in Egypt by day over the preceding two months. Then, the history -- from 1979 to 2010 -- of unrest in every country in the world could be searched and compared to it. This could be done using a rolling, 60-day window. Take Afghanistan, for instance: The timeline of that country's unrest from Jan. 1, 1979, to Mar. 2, 1979, could be compared against the selected period in Egypt. Then, Afghanistan's unrest from Jan. 2, 1979, to Mar. 3, 1979, could be analyzed -- and so on. This would allow researchers to identify past periods in history that are most similar to Egypt's tumultuous two months.

Then, the top two most similar historical periods could be selected, and the average of what happened within them -- what events transpired, how, and in what time frame -- could be computed as a possible forecast of what might happen in Egypt in the immediate future.

Figure 1 here shows the results of this exact process, plotting the relative intensity of unrest on the y-axis. To the left of the vertical black bar is Egypt's selected two months in red, overlaid on top of one of the most similar periods of past world history: events in Sweden, from Oct. 4, 2010 through Dec. 3, 2010, which are shown in green. Sweden's unrest during this period stemmed from the (ultimately disproved) anti-Semitic swastika attack against politician David von Arnold Antoni, the Dec. 7 arrest of Julian Assange, and the Dec. 11 Stockholm bombings. Despite entirely different circumstances, the two countries show nearly identical relative intensities of unrest over their respective 60-day periods.

To the right of the vertical black bar is what then happened in Sweden over the two months after Dec. 3, 2010, compared with what ended up happening in Egypt in the two months after Jan. 27, 2011. Immediately it is clear that, while there are significant differences, the trend lines bear strong resemblance to each other. If an analyst had been running this system in 2011, he or she could have used the Sweden data to forecast what was going to unfold in Egypt: that is, roughly two weeks of notable unrest, then a return to (relative) levels of calm.

Figure 1 - Sweden 10/4/2010 - 12/3/2010 (green left of black line) and 12/3/2010 - 2/1/2011 (green right of black line), compared with Egypt (red). (Click to enlarge)

In other words, by searching for periods of history all over the world similar to what is happening now in a single country, one can look at what happened after each of those periods as a measurable forecast of what might happen in the target country in the future. And as this graph indicates, history indeed seems to repeat itself -- at least as seen through the eyes of the world's news media. (Those mathematically inclined can read the rest of the details of the forecasting, including the underlying code.)

Replicating this process for Ukraine starting on Feb. 22, 2014 (the day then-President Viktor Yanukovych fled the country), and averaging the two most similar historical periods -- from Turkey and Lebanon -- ultimately yields the following forecast (green in Figure 2) compared with what actually happened in Ukraine over the following two months (red). This is particularly noteworthy because, unlike with Egypt, the post-period here exhibits far more complex behavior, breaking from traditional "media fatigue" and other journalistic effects.

Figure 2 - Ukraine (red) compared with average of Turkey and Lebanon (green) in the following 60 days, after the correlated time periods. (Click to enlarge)

Of course, where there is forecasting there is a need to see whether or not it is accurately predicting events. So in addition to timelines, GDELT has developed a daily, visual update that shows the current state of the world, letting analysts watch events as they unfold and compare them to projections. In collaboration with the U.S. Institute of Peace, the new GDELT Global Dashboard offers a rolling, 180-day map of conflict and protests across the globe. The Islamic State's march across Iraq is instantly visible, civil war in Ukraine bursts into view, and the clustering of protests and violence in Nigeria also leaps from the screen. This map is updated every morning, offering both a clickable layer of the latest events and a six-month animated context of what preceded the present state of affairs.

Figure 3 -  The GDELT Global Dashboard. (Click to enlarge)

For thousands of years philosophers and historians have argued that the world moves in broad, predictable patterns that provide insight into the future. Humanity has finally reached a critical juncture where it has sufficient data and computing power to begin to tease apart these patterns and understand them in a way that was simply impossible before. Forecasting the future of Egypt or Ukraine, by looking to all of the history of the last three decades, required more than 2.5 million comparisons -- yet it took Google's BigQuery system just 2.5 minutes to run them. 

Whether the patterns identified capture some kind of new mathematical equation that governs all of human life or -- perhaps far more likely -- a more precise mathematical definition of how journalism shapes our understanding of global events, they demonstrate the unprecedented power of the new generation of "big-data" tools. The world may not quite be at the level of Isaac Asimov's psychohistory, able to predict global events like meteorologists do the weather. But it is now possible to glimpse the future -- and better understand how to shape it.


Democracy Lab

The Political Magic of Roads

Is real democracy possible without highways?

A friend was asking me about my recent trip to Rwanda. What, he wanted to know, surprised me the most?

"I think it was the roads," I said. "The roads were amazing."

"I'll bet," he said. "They must have been a mess, right?"

No, actually, just the opposite. I explained that the streets in Kigali, the capital, are smoothly surfaced, with nary a pothole in sight. Well-crafted rain gutters and zebra-striped curbs mark the edges of each roadway. Blinking warning lights are embedded in the road surface at corners. Traffic lights don't just show green for go and red for stop; they're also equipped with digital clocks, showing how long until each light changes color. The wealthy suburb of Washington, D.C., that I call home would be happy to have such roads.

Nor is this a privilege of Rwanda's capital. When we set out on a trip to the city of Goma, just over Rwanda's border with the Democratic Republic of the Congo, the highway that took us there was just as impressive. I barely noticed a bump in three hours. My subsequent travels around the country confirmed that this was no fluke. Rwanda is a nation of remarkable roads.

This discovery struck me with particular force because of one of my other recent trips to Africa. In December I visited Mali, the West African country that nearly fell apart two years ago when Tuareg separatists in the north, aided by jihadi allies, decided to break away. The drive from Mopti, in central Mali, to the northern city of Timbuktu covers just 250 miles -- but this rather modest journey took us nine bone-shattering hours (not counting stops).

It's a roller-coaster dirt track that rarely permits speeds higher than 40 miles per hour. In order to avoid the worst we often swung off the "highway" onto barely visible trails through the bush that paralleled it. By the time we arrived in Timbuktu, I found myself sympathizing with the separatists: Why should northerners feel any loyalty to their compatriots in Bamako, the capital in the south, if the government can't even be troubled to connect the two parts of the country with a proper road?

The comparison is compelling. Mali is a democracy that boasts free elections and a pluralistic press but is also plagued by miserable governance and entrenched corruption that continue to hamper development. Rwanda is an autocracy that tolerates little dissent but has a remarkable record of delivering public services to its citizens (including infrastructure). Both countries have traumatic histories of poverty and ethnic division. But while Mali continues to struggle, Rwanda is on the move.

That good roads have a positive economic effect seems like a no-brainer. But my travels have made me inclined to think that we tend to underestimate the political effects of transport infrastructure. In Rwanda, decent roads stand for the official commitment to provide everyone with equal access to the fruits of development -- concrete evidence, if you will, of the determination to overcome the ethnic divides that led the country into mass slaughter just two decades ago. Every part of the country is relatively close to a good road; no group is excluded. (Nor do you have to have a car to get around; members of Rwanda's growing middle class can simply hop on one of the country's ubiquitous minibuses.) Northern Malians can only dream of such conditions.

Roads don't just enable the movements of goods; they also enable the flow of ideas. Christianity in its present form probably wouldn't exist today if it weren't for the extraordinary Roman road network that enabled the Apostle Paul to transmit his teachings across the empire. The establishment of the U.S. interstate highway system in the 1950s completed the work of national unification that started with the construction of transcontinental railroads in the 19th century (and pushed the country into the modern era of indistinguishable suburbia and big-box stores).

As such, roads are also crucial ingredients of state-building. Soon after toppling the Taliban in 2001, the U.S.-led coalition that occupied Afghanistan set out to rebuild Highway 1, the ring road linking Kabul with the country's major urban centers (including Kandahar, the Taliban's unofficial capital). The idea was to restore a sense of unity to a country that had virtually fallen apart during the long years of civil war. Taliban insurgents immediately vowed to sabotage the project for just the same reason. Today, the dismal story of Highway 1, which is falling apart after $4 billion of Western investment, offers a perfect microcosm of Afghanistan's roller-coaster struggle to reinvent itself. Afghanistan isn't unusual in this respect. Take a look at many failed states around the world and you'll probably be struck by how many of them have bad (or no) roads.

Of course, there's a big chicken-or-the-egg question here: A country's governing class probably won't be capable of building proper roads unless it's fairly competent to begin with. That's certainly the case with the ruling Rwandan Patriotic Front (RPF) of President Paul Kagame, an ascetic former guerilla leader whose obsessive-compulsive-disorder approach to rule has even prompted him to ban plastic bags.

The Tutsi-dominated RPF, whose members spent decades in exile in countries adjacent to their homeland, took power after the collapse of the Hutu Power government that implemented the mass slaughter of mostly Tutsi Rwandans in 1994, and the memory of that trauma gives an extraordinary urgency to the government's efforts to shape a national identity that transcends old divides. (And building roads does play a part in that larger program. The authorities often send Rwandans to work on local roads as part of government-sponsored job creation schemes or compulsory "community service" programs.)

Despite what some of its defenders say, Rwanda is no democracy; it's a tightly organized one-party state. (Incidentally, it's a company from Singapore, the one-party state that Kagame most often cites as his model, that designed the urban plan for Rwanda's capital, including its remarkable streets.) Even so, if you had to ask me whether Mali or Rwanda is more likely to achieve a prosperous, fully functioning democracy one day in the future, I'd probably have to pick the latter. Can you really have a democracy when one half of the country doesn't feel like it is part of the rest? Can you have a democracy without citizens who are connected on the most elementary physical level?

The Internet and the increasing sophistication of virtual worlds may one day change this equation. But don't hold your breath. For now, tarmac is the test.

Visions of America/UIG via Getty Images