Doomsday Preppers

At a new center in Cambridge, a philosopher, an astronomer, and a software pioneer are looking for ways to save humanity from itself.

"Sometimes I feel I'm irrationally optimistic," says Huw Price. This is, perhaps, an unlikely statement for the co-founder of an organization dedicated to studying scenarios for the end of life as we know it. Price, an Australian philosopher known for his work on time and cosmology, is working to build the Centre for the Study of Existential Risk (CSER) -- a proposed new think tank at Cambridge University with the ambitious goal of "ensuring that our own species has a long-term future." A less cheery way of putting it is that the center will study possible ways that humanity is planting the seeds of its own destruction. "One of the problems we need to deal with is to find ways of coping with our own optimism," Price says.

To that end, he has partnered with two thinkers who couldn't really be described as glass-half-full guys. Martin Rees, a Cambridge astrophysicist who serves as Britain's royal astronomer, is the author of Our Final Century, a 2002 book predicting that, due to a lethal combination of possible natural and man-made catastrophes, our civilization has only a 50 percent chance of surviving through the year 2100. (In the United States, the book was published as Our Final Hour, because, Rees jokes, "Americans like instant gratification.") A veteran of the nuclear disarmament movement, he has also predicted that by 2020, "bioterror or bioerror will lead to 1 million casualties in a single event."

Rees seems positively cautious compared with the third member of the unlikely trio, Estonian computer programmer and technology theorist Jaan Tallinn, one of the key developers of Skype and, before that, the file-sharing service Kazaa. It was Tallinn who inspired Price to start the center while the two were splitting a cab at a conference in Copenhagen last year by stating matter-of-factly that he believes he has a greater chance of being killed by an artificial intelligence-related accident than by cancer or heart disease -- the leading causes of death for men in his demographic. After all, every advance in technology makes these natural causes less likely and an AI disaster more likely, he explained.

CSER's founders aim to make scientists and developers of new technologies think more about the long-term consequences of their work. They also make the somewhat radical suggestion -- in scientific circles -- that new scientific knowledge is not always worth acquiring. Research on developing more deadly strains of the influenza virus might be one example. "We're trying to embed people whose job it is to think about risks into technology development teams in order to raise the consciousness of people in technology about potential risks," Price says. They hope that the message might resonate more coming from figures like Rees and Tallinn, whom nobody could accuse of Luddism.

The center is still in its fundraising stage, but has already attracted a list of high-profile advisors from a variety of fields, including their Cambridge colleague Stephen F. Hawking, the renowned astrophysicist. Depending on the level of funding they receive, Price says he imagines the center will consist of more than a dozen postdocs working with faculty advisors and will serve as a kind of clearinghouse for research on catastrophic risk from specialists around the world. "People interested in these issues tend to be very scattered in different disciplines and geographically," Price explains.

Some of the eclectic cast members who have already signed up range from development economist Partha Dasgupta -- whose work has explored the value society ought to place on future lives, as opposed to current lives, in the context of disasters like climate change -- to Nick Bostrom, the philosopher of technology known for posing such questions as the Matrix-esque, "Do we live in a computer simulation?"

Some of the risks the center will tackle are well known and frequently discussed -- nuclear war, for instance. "The threat of nuclear annihilation is only in temporary abeyance," says Rees. "We were lucky to get through the Cold War without a catastrophe. Even though the risk of tens of thousands of bombs going off now is less than it was then, we can't rule out a shift in the next 50 years that could lead to a new standoff, handled less well than the Cold War was."

Others subjects the center hopes to tackle are a bit more exotic, such as Tallinn's fears about hyperintelligent machines. Tallinn's ideas build on the work of past theorists like the pioneering computer scientist I.J. Good, who predicted in the 1960s that once machines became intelligent enough to reproduce themselves, it would trigger an "intelligence explosion" that would leave human beings in the dust. "The first ultraintelligent machine is the last invention that man need ever make," Good wrote in 1965. Futurists like Ray Kurzweil and Vernor Vinge developed the idea with their concept of a technological "singularity" -- the point at which artificial intelligence develops so quickly that the consequences become nearly impossible to predict. Tallinn believes there is a "double-digit" chance of the singularity occurring this century.

So what makes ultraintelligent machines so dangerous? Couldn't we coexist with our new robot cohorts? Well, for one thing, as Tallinn points out, it's not at all clear they would need to keep us around. "The idea of robots having their own society is just hopeless anthropomorphic," he says. "They are not part of biological evolution. So you might not get a society of machines. You might just get a single machine that bootstraps itself into some really unseen level of intelligence. We're talking about living on a planet whose environment we no longer control, just as other species no longer control their environment right now."

Rebellious machines that turn on humanity are a staple of pop culture, from the 1921 Czech play that coined the word "robot" to modern classics like The Terminator and Battlestar Galactica. The motivation behind Tallinn's work with the center is to discuss such scenarios outside the realm of science fiction and encourage those who work in technology to take them seriously. "I'm not advocating refraining from technology development," he says. "But as tech gets more powerful, we need to consider all the consequences -- positive and negative -- before we proceed. But we can't just be permanently techno-optimist or techno-utopian."

As for Rees, what really keeps him up at night isn't nuclear war or robot uprisings or even asteroid impacts, which he sees as a straightforward problem with a "quantifiable risk which we can say is worth a few hundred million dollars to mitigate." It's the risks we haven't even thought of yet -- the unknown unknowns, if you will.

"The financial crisis was an example of something that no one predicted that went global because of interconnectedness in the world," Rees says. "It's a metaphor for what might happen with other kinds of breakdowns due to error or terror." The work of the center will be to separate the risks that are worth worrying about from those that can be left to the sci-fi authors. "We're probably talking about things that have a less than 50 percent chance individually. But that's true of the fire insurance on your house. Chances are your house won't burn down, but it's worth taking precautions to minimize that risk."

The founders also hope to get an early start in talking about these risks, as it might not be easy to convince policymakers to take them seriously. "If you look at the kinds of influence serious climate scientists have on policy, it isn't very encouraging," Price says. If scientists can't get political leaders to take serious action on an ongoing problem whose worldwide effects are easily noticeable and quantifiable, it's hard to imagine they'll have better luck warning of the risks of, say, hyperintelligent machines.

Then there's the question of whether encouraging more fear in the public is really a worthwhile course of action. From the serious (terrorist attacks) to the depressingly mundane (household products causing cancer) to the ridiculous ("Razor blades in your child's Halloween candy! News at 11."), we're constantly bombarded by things to be afraid of. But Price says this only makes it more important that we prioritize what we should be worried about.

"People tend to worry about the wrong things and make bad judgments about risk," he says. "People worry that their kids might be grabbed by strangers on the way to school, so they won't let them walk to school. Instead, they drive them to school and put them at much higher risk of injury. As a community, given that these kinds of risks are on the table, we need to be better at dealing with them."

A robot uprising might not be high on the list right now, but you can never be too careful.



Longform's Picks of the Week

The best stories from around the world.

Every weekend, Longform highlights its favorite international articles of the week. For daily picks of new and classic nonfiction, check out Longform or follow @longform on Twitter. Have an iPad? Download Longform's new app and read all of the latest in-depth stories from dozens of magazines, including Foreign Policy.

Aarushi Talwar Murder: The Inside Story of India's Most Controversial Trial
Shree Paradkar • The Toronto Star

On the killing of a 13-year-old girl and how her parents came to be charged with murder.

Sex. Illicit affairs. Murder. Indian media, which combines British tabloid sensibility with U.S. cable's cutthroat competitiveness, snapped it up and fed it to a gossip-hungry audience, catapulting the crime to the top of the news cycle and making Aarushi a household name.

"India's JonBenet Ramsey case??" asked a Time magazine headline.

Four-and-a-half years, one state police force, two federal investigative teams, two sets of suspects, five arrests and countless fumbles later, Aarushi's murder remains unsolved.


Hackers in China Attacked the Times for Last 4 Months
Nicole Perlroth • New York Times

The paper of record reports on a battle of its own.

The mounting number of attacks that have been traced back to China suggest that hackers there are behind a far-reaching spying campaign aimed at an expanding set of targets including corporations, government agencies, activist groups and media organizations inside the United States. The intelligence-gathering campaign, foreign policy experts and computer security researchers say, is as much about trying to control China's public image, domestically and abroad, as it is about stealing trade secrets. 

Adam Berry/Getty Images

New Old Libya
Robert Draper • National Geographic

In the wake of revolution, Libyans envision their future.

Yet on the desert slog to rediscovery, flag-waving offers only the mirage of a shortcut. As Sury acknowledged, Libya's rebuilding "starts at zero." The terrorist attack last September casts a dark shadow over Libya's attempts to increase stability and rebuild its government. Whether the 30,000 Libyans who protested against militias ten days later constitute a better predictor of Libya's future, it is too early to say. In ways both obvious and insidious, Libya remains half-blinded by its former dictator's heavy hand. Now, like the statue in the wooden box, it awaits its future in an unforgiving light.

John Moore/Getty Images

For 40 Years, This Russian Family Was Cut Off From All Human Contact, Unaware of World War II
Mike Dash • Smithsonian

A team of scientists make an accidental discovery.

The sight that greeted the geologists as they entered the cabin was like something from the middle ages. Jerry-built from whatever materials came to hand, the dwelling was not much more than a burrow-"a low, soot-blackened log kennel that was as cold as a cellar," with a floor consisting of potato peel and pine-nut shells. Looking around in the dim light, the visitors saw that it consisted of a single room. It was cramped, musty and indescribably filthy, propped up by sagging joists-and, astonishingly, home to a family of five:

The silence was suddenly broken by sobs and lamentations. Only then did we see the silhouettes of two women. One was in hysterics, praying: 'This is for our sins, our sins.' The other, keeping behind a post... sank slowly to the floor. The light from the little window fell on her wide, terrified eyes, and we realized we had to get out of there as quickly as possible.


The Republic of Port Said
Evan C. Hill • Foreign Policy

How a local insurrection is challenging the Muslim Brotherhood's control of Egypt.

What began as a soccer rivalry between Cairo and this city of 600,000 at the mouth of the Suez Canal sparked a tragic stadium riot in February 2012 and has now grown into a miniature insurrection. After a court's decision to hand down 21 death sentences against Port Said soccer supporters for their role in the riot, outraged men allegedly rushed toward the prison and tried to shoot the inmates out, resulting in deaths on both sides.

The violent reaction has become the most critical test of Morsy's ability to steer the nation since his election last June, and the president declared a curfew and state of emergency along the entire Suez Canal in an effort to contain the unrest. But Morsy's attempt to restore order only highlighted his precarious control over Egypt and its institutions, and the canal cities reacted with outright defiance.

STR/AFP/Getty Images

For daily picks, check out Longform.org or download the Longform App.