Predicting the Future Is Easier Than It Looks

Nate Silver was just the beginning. Some of the same statistical techniques used by America's forecaster-in-chief are about to revolutionize world politics.

Before Billy Bean and the sabermetric revolution upended baseball and ushered in a new era of statistically driven baseball analysis, old-timers insisted that the young eggheads and their spreadsheets were no match for a time-worn scout. Experience, gut feeling, and a sense of the intangible qualities that make up a quality prospect -- these were the things that the old guard argued could never be captured by an Excel spreadsheet, let alone a statistical model.

By and large, they were wrong, and Billy Bean's scrappy Oakland Athletics squads showed that the so-called eggheads could see further into the future and with greater clarity than had previously been thought possible.

The same statistical revolution that changed baseball has now entered American politics, and no one has been more successful in popularizing a statistical approach to political analysis than New York Times blogger Nate Silver, who of course cut his teeth as a young sabermetrician. And on Nov. 6, after having faced a torrent of criticism from old-school political pundits -- Washington's rough equivalent of statistically illiterate tobacco chewing baseball scouts -- the results of the presidential election vindicated Silver's approach, which correctly predicted the electoral outcome in all 50 states.

That kind of nay-saying -- epitomized by Joe Scarborough's comments that anyone who thinks the presidential race was anything but a toss-up is a joke -- is exactly what the baseball establishment said to Billy Bean before this approach became popular in professional athletics, where it has spread far beyond a few losing baseball teams to a wide range of sports. And it is exactly what some parts of the political world said to the poll aggregators even during the most recent presidential election.

But the statistical revolution in presidential politics begs the question: Why can't the same predictive tools be applied to world politics? The amount of data that exist on world politics is enormous. Perhaps it is time to focus on making more sense of it.

The first real attempt to ask whether conflicts could be predicted can be found in Quincy Wright's 1942 magnum opus A Study of War. Interestingly, he believed short-term forecasting should be based on public opinions rather than economic and political indices. But if forecasts were to be based on indices, Wright wrote, they "should have a value for statesmen, similar to that of weather maps for farmers or of business indices for businessmen ... Such indices could be used not only for studying the probability of war between particular pairs of states but also for ascertaining the changes in the general tension level within a state of throughout the world."

But on Nov. 8, Jay Ulfelder, himself a forecaster, tried to throw a bucket of cold water on this idea, arguing in Foreign Policy why it is impossible to have a "Nate Silver" in world politics. (The fact that Nate Silver's name can now be used as a metaphor for a perfect predictive model is one indication of his enormous success and popularity.)

First, Ulfelder tells us that election forecasting is the leading edge of statistical forecasting and that the things that foreign policymakers want to know are far from that edge. Perhaps, but this is not transparently clear. Today, there are several dozen ongoing, public projects that aim to in one way or another forecast the kinds of things foreign policymakers desperately want to be able to predict: various forms of state failure, famines, mass atrocities, coups d'état, interstate and civil war, and ethnic and religious conflict. So while U.S. elections might occupy the front page of the New York Times, the ability to predict instances of extreme violence and upheaval represent the holy grail of statistical forecasting -- and researchers are now getting close to doing just that. In 2010 scholars from the Political Instability Task Force published a report that demonstrated the ability to correctly predict onsets of instability two years in advance in 18 of 21 instances (about 85%), including the prediction of instability in Iran in 2004 and Côte d'Ivoire in 2002, for example. None other than Jay Ulfelder was a co-author of this study, so he may just suffer from excessive modesty.

Just within the environmental conflict realm, a recent report by the Army Environmental Policy Institute lists no fewer than twelve ongoing projects that touch on some aspect of forecasting. These include the USAID's Famine Early Warning System which tracks and predicts food insecurity around the world  as well as the Climate Change and African Political Stability project, housed at the Robert S. Strauss Center at the University of Texas at Austin. Outside of the environmental arena, there are more.

Forecasting models need reliable measures of "things that are usefully predictive," Ulfelder notes. Well, sure.  Does this mean that reliability is at issue? Or that we are using data that are not "usefully" predictive? This is a curious claim, especially in light of the controversial nature of polls. Indeed, there exists five decades worth of literature that grapples with exactly those issues in public opinion.  Take the recent U.S. election as an example. In 2012 there were two types of models: one type based on fundamentals such as economic growth and unemployment and another based on public opinion surveys. Proponents of the former contend that that the fundamentals present a more salient picture of the election's underlying dynamics and that polls are largely epiphenomenal. Proponents of the latter argue that public opinion polling reflects the real-time beliefs and future actions of voters.

As it turned out, in this month's election public opinion polls were considerably more precise than the fundamentals. The fundamentals were not always providing bad predictions, but better is better. Plus there is no getting around the fact that the poll averaging models performed better. Admittedly, many of the polls were updated on the night before the election, though Drew Linzer's prescient posted predictions last June that held up this November. To assess the strength of poll aggregation, we might ask how the trajectory of Silver's predictions over time compare with the results, and there are other quibbles to raise for sure. But better is better.

When it comes to the world, we have a lot of data on things that are important and usefully predictive, such as event data on conflicts and collaborations among different political groups within countries. Is it as reliable as poll data? Yes, just so, but not more. Would we like to have more precise data and be able to have real-time fMRIs of all political actors? Sure, but it is increasingly difficult to convincingly argue that we don't have enough data.

Let's consider a case in which Ulfelder argues there is insufficient data to render a prediction -- North Korea. There is no official data on North Korean GDP, so what can we do? It turns out that the same data science approaches that were used to aggregate polls have other uses as well. One is the imputation of missing data. Yes, even when it is all missing. The basic idea is to use the general correlations among data that you do have to provide an aggregate way of estimating information that we don't have. We know enough about how other things are related to GNP that we can figure out reasonable estimates of what range it falls into in places where it is not observed. The CIA estimates North Korean GDP per capita at $1800 for 2011, based on extrapolations, growth rates estimations, and inflation. Our Duke imputations are based on lots of other data, but no data whatsoever for North Korean GDP, and we have it at about $1700, perhaps close enough for government work.

Our point is that collecting data in new and exciting ways has changed the nature of political forecasting. While the Twitterverse might have been agog over the daily release of the Gallup tracking poll, the real story of the election was playing out elsewhere. The firm Latino Decisions did not start frequent polling until 2010 (it was founded in 2007) but conducted more than 60,000 questionnaires during this year's election and found that the Latino vote was going to be overwhelmingly in favor of the Democrats. One need only glance at this year's exit polling to know that the Latino vote was crucial to President Obama's reelection. So by focusing on Gallup -- whose results were wildly off the mark anyway -- many mainstream pundits ignored a wealth of other, perhaps more important information.

Ulfelder and others argue that the world needs a large amount of data from a large "swath" of history to effectively develop good models, but this kind of thinking often fails to reflect the mechanics of a statistical forecast. We've had elections in this country for more than two centuries at last count, but very few of the models that were applied to predict the 2012 election employed data on each of them. Some structural models only used data on a few years, and the polling data obviously doesn't go back very far.

There is a tradition in world politics to go either back until the Congress of Vienna (when there were fewer than two dozen independent countries) or to the early 1950s after the end of the Second World War. But in reality, there is no need to do this for most studies. Typically, data is needed only for the current political era, which might date to 1989 or even just through the current century. Don't be misled by claims that there "won't be enough cases to analyze" if you use these "shortened" time frames. This makes the assumption that you have to analyze annual data, an assumption that is just blatantly false, even if it has been standard operating practice in quantitative world politics for decades. Not only are there techniques available for analyzing data on a much smaller time scale (days, weeks, or months), but if we use them, we are likely to get closer to those elusive variables that the policymakers lust after. And generally, the data have to be tortured before they surrender to annual formats. Consider, for instance, a coup on January 1 in 2010 and another one on December 31st. These two coups occurred in the same calendar year, but did they actually occur at the same instant?

Ulfelder tells us that "when it comes to predicting major political crises like wars, coups, and popular uprisings, there are many plausible predictors for which we don't have any data at all, and much of what we do have is too sparse or too noisy to incorporate into carefully designed forecasting models." But this is true only for the old style of models based on annual data for countries. If we are willing to face data that are collected in rhythm with the phenomena we are studying, this is not the case. For example, Thailand became considerably more democratic in July 2011 as a result of Yingluck Shinawatra winning a landslide election and successfully forming a coalition government to replace a government established by a coup d'état. There is no need to assume this change to a more democratic form of government applies to the entire year of 2011, since we know it didn't really characterize Thailand during the first half of the year. We have data and techniques that can deal with monthly or even daily data. Whether the data are too noisy to make use of is an empirical question. Our hunch is that clever data scientists will find a way to make these data useful.

Consider thyroid cancer. According to the National Cancer Institute, thyroid cancer has an incidence of about 6 in 100,000. It is a rare event, but we know quite a lot about it, even to the level of making preventive prescriptions. The rareness does not prevent us from learning about its occurrence, how to treat it, and even how to best avoid this cancer.

Don't get us wrong: Better data is always better. But we actually have a lot of data. We don't want to argue that forecasts about world politics are as precise as we observed for the 2012 presidential elections. And we agree with Ulfelder that they are not. But we are not so pessimistic to think that forecasting of certain kinds of events cannot in principle be solved by a combination of statistical approaches and data by clever investigators. David Rothschild, David Pennock, and a team of about 30 others (then at Yahoo; now mainly at Microsoft Research) predicted 303 electoral votes for Obama on February 15th, 2012 not by building a model of the election, but as they put it, by being "the mother of all prediction engines, period." There is considerable effort underway, beyond Microsoft Research, to build predictive models that relate to world politics.

Better, of course, will always be better. But there is room for seeing the water in the glass, not just how big the glass is.



Throwing Rice

Barack Obama was right to defend Susan Rice from the GOP attacks. She's got the chops to be secretary of state.

In his first press conference since the election, on November 14, President Barack Obama made a forceful defense of his ambassador to the United Nations, Susan Rice, who increasingly appears to be the favored candidate to replace Hillary Clinton as secretary of state. The public display of support caught some analysts by surprise, both because they assumed the position was likely to go to someone else -- presumably Sen. John Kerry -- and because they thought her role in the Benghazi tragedy, repeating administration talking points that later turned out to be false, would make Senate confirmation extremely difficult.

But by any reasonable measure -- qualifications, record, and capacity to do the job effectively -- the case for Susan Rice is strong. It is therefore appropriate that President Obama stood by his ambassador -- both in the second presidential debate and in his news conference

Yesterday, the president was clear: "I don't think there's any debate in this country that when you have four Americans killed, that's a problem, and we've got to get to the bottom of it and there needs to be accountability. We've got to bring those who carried it out to justice. They won't get any debate from me on that," he said. "But when they go after the U.N. ambassador, apparently because they think she's an easy target, then they've got a problem with me." But a number of Republicans have gone after her nonetheless. As Sen. Lindsey Graham said in response, he has "no intention of promoting anyone who is up to their eyeballs in the Benghazi debacle."

Despite Graham's and others' best efforts to use Rice as a billiard ball to keep the Benghazi story alive, there is no evidence that she is "up to her eyeballs" in anything -- let alone the Benghazi fiasco, which she had virtually nothing to do with. What we do know, is that she did precisely what any U.N. ambassador is supposed to do: convey the administration's position based on the information that is available at the time.

There is no doubt that the attacks in Benghazi occurred against the backdrop of the inflammatory anti-Muslim video, which had global resonance and required a global response -- appropriate for the U.N. ambassador. Moreover, her specific comments on "Face the Nation" and other Sunday talk shows were based on an intelligence timeline that had been provided to her by the administration and which reflected what was known at the time. As the Washington Post reported on Oct. 19, Rice's talking points had actually been prepared for the House Intelligence Committee, which evidently found them credible enough.

If anything, Rice and the intelligence community might be faulted for having to go public with what was a tentative intelligence analysis. Perhaps the administration can be faulted for putting anyone on the national talk shows to discuss the attack while the intelligence was itself evolving -- seemingly as they spoke. But if they hadn't, the outcry would have been in the other direction: people would have been demanding to know why the administration wasn't sharing all the information it had.

As the president said yesterday: "[F]or them to go after the U.N ambassador, who had nothing to do with Benghazi and was simply making a presentation based on intelligence that she had received, and to besmirch her reputation, is outrageous." Whether or not Obama selects Rice as his next secretary of state (and, for the record, he said that he hadn't made a determination yet), it is reassuring to see a commander-in-chief stand up for his representative at the United Nations and for the professionals in the intelligence community working with the best information they had.

Republican senators, for their part, shouldn't judge any highly qualified potential nominee before even listening to the case for their selection. The tradition of deference to presidents as they select their cabinets is an important one -- as is the practice of investigation, and advice and consent. First and foremost, that process should be geared toward determining if Rice is qualified to be secretary of state. No question, Susan Rice's record has plenty of room for serious questions and opportunities for her to explain. But to assert even before a nominee is announced that the are "unqualified" -- as Sen. John McCain did this week -- is not becoming of the great traditions of the Senate.

The answer to this is a resounding "yes": Rice's distinguished diplomatic career includes a stint on the National Security Council in the 1990s; assistant secretary of state for African affairs (1997-2001), where she oversaw the management of 43 U.S. embassies and 5,000 Foreign Service personnel; and eight years as a senior fellow at the Brookings Institution. Since 2009, the former Rhodes Scholar and management consultant has also served effectively as ambassador to the United Nations -- a position previously held by President George H.W. Bush, Secretary of State Madeline Albright, and the late Ambassador Richard Holbrooke -- who also would have been a strong contender for this position. On paper, then, Rice is as at least as qualified for the position as most of her recent would-be predecessors.

Moreover, Rice has already proved that she is a player on the global stage. She was largely responsible for building international consensus around the U.N. resolutions that enabled the Libyan intervention. Rice has also rallied global support for strong and sustained pressure on Iran, and has taken on Russia over the Syrian crisis, though her success in Libya may have made failure in Syria inevitable. To this point, she recently indicated a balance between interventionist ideals -- which she successfully achieved in helping to rally the U.N. Security Council to pass essential resolutions that paved the way for the end of the Qaddafi regime in Libva -- and realism, stating to the Foreign Policy on September 25 of Syria that: "I'm not of the view that this is a circumstance in which external military intervention is wise for the United States or others," she said. "I don't think that [the Responsibility to Protect] was ever a cookie-cutter response that was going to automatically be implemented everywhere things went badly."

If selected, Rice would provide both continuity in a critical moment of transition and the eyes of a foreign policy practitioner who came of age in an era of new global challenges. Crucially, Rice has the ear of the president -- and of other world leaders through her work at the United Nations.

While other likely candidates are equally credentialed and would likely do the job well, it is crucial that the secretary of state be immediately understood when engaging in public and private diplomacy overseas as someone that has a direct line to the president. History tells us that when the secretary of state is not in the immediate decision-making loop, this can make it more difficult to ensure the priority of diplomatic approaches to asserting the national interest. This is not to say other nominees would not enjoy this advantage -- but it is one that Rice would begin with from day one. Moreover, because of that access, she would also be in a strong position to represent the interest of the Foreign Service when interacting with other key agencies of government. She would thus be well-equipped to return America's diplomatic corps to its proper role after a decade in which military influence dominated decision-making.

Of course, Obama may yet select a wildcard candidate as his next secretary of state. And there remains a strong case to be made for other serious candidates for this position -- particularly Sen. John Kerry -- all of whom would serve America with distinction. But Rice merits the respect that her service and qualifications deserve and it is to Obama's credit that he has stood by her and recognized her achievements. It is entirely appropriate that her name is under serious consideration.