DARPA’s Crisis Early Warning and Decision Support System

The International Studies Review just published a piece by Sean O’Brien entitled “Crisis Early Warning and Decision Support: Contemporary Approaches and Thoughts on Future Research.” Sean outlines the latest attempt by the US military (ie, DARPA) to develop a crisis forecasting tool. This time, the platform is called ICEWS for Integrated Crisis Early Warning System.

O’Brien gives a brief overview of recent efforts in this space including Bueno de Mesquita’s Policon and Senturion forecasting systems, which are said to be 90%+ accurate. That said, O’Brien notes that Mesquita himself acknowledges that “he is not exactly sure how to interpret this accuracy claim since most of the reported assessments he has were not explicit about how accuracy was measured.” Incidentally, I like how this rather important qualifier is buried at the bottom of a footnote.

Some of Policon’s/Senturion’s supposedly accurate predictions had to do with questions like:

  • What policy is Egypt likely to adopt towards Israel?
  • What is the Philippines likely to do about US bases?

Keep in mind that millions of dollars were spent on these sophisticated systems and yet I can’t help but think that paying some experts on Egypt and the Philippines a few thousand dollars would have more or less accomplished the same task. Apparently, Senturion accurately predicted the deteriorating disposition of Iraqis toward US forces. Really? Shocking, who would have expected Iraqi public opinion to shift? Yes, that was sarcasm.

O’Brien also references a forthcoming study by Ward, Greenhill and Bakke, which “delivers a serious blow to the predominant way in which most conflict models are evaluated using statistical significance.” These include the predictive models developed by Fearon and Latin (2003) and Collier and Hoeffler (2004). Ward et al. show that these models predict few if any civil war cases at a reasonable probability cut off of 50%. In fact, the Fearon and Latin model “does not even appear to generate a probability of greater than 30%.” In sum, Ward et al. conclude that we cannot correctly predict over 90% of the cases with which our models are concerned.

Many of the most interesting, policy-relevant theoretical questions are also the most complex, nonlinear, and highly context-dependent. They demand consideration of hundreds of massively interacting variables that are difficult to measure systematically and at a level of granularity consistent with the theory. In such cases it is at best impractical and at worst impossible to apply standard regression techniques within the context of a Large N study, short of invoking unreasonable, oversimplifying assumptions. This may in part account for contradictory findings in the literature relative to the validity of alternative theoretical claims.

So lets keep in mind that previous “breakthroughs” have since been largely discounted.

ICEWS phase one of three consisted of a competition between different groups to successfully predict events of interest (EoI) on a set of historical data. The most successful team was Lockheed Martin-Advanced Technology Laboratories (LM-ATL) in cooperation with a number of established scholars and industry partners. The team integrated and applied six different conflict modeling systems, including:

  1. Agent-based models drawn from Barry Silverman’s Factionism and Ian Lustick’s Political Science-Identity (PSI) computational modeling platforms. The latter is created with “agents representing population elements of various ethnic ⁄ political identities organized geographically and in authority structures designed to mirror the society being studied.”
  2. Logistic regression models developed by Phil Schrodt and Steve Shellman, which use “macro-structural and event data factors commonly analyzed in the academic literature.” Shellman’s approach uses a Bayesian statistics model.
  3. Geo-spatial network models built by Michael Ward, which uses “structural factors, event counts, and various types of spatial networks—trade ties, people flows, and ‘‘social similarity’’ profiles—that embody potential EOI co-dependencies between proximate countries.”
  4. “A final model was developed by aggregating the forecasts from the above mentioned models using Bayesian techniques.

I’m particularly interested in the use of Agent-based models (ABM) for conflict analysis. O’Brien references a very interesting project at Virginia Tech which I was unaware of:

Scholars at Virginia Tech have already developed a 100 million agent simulation that includes synthetic versions of many American citizens, and plan to expand to 300 million agents this year (Upson 2008). Each synthetic agent has as many as 163 variables describing age, ethnicity, socio-economic status, gender, and various attitudinal factors. The simulation is used to assess how different types of pandemics could spread across the United States under different scenarios.

Patrick Philippe Meier

6 responses to “DARPA’s Crisis Early Warning and Decision Support System

  1. Very interesting post Patrick. I’m very skeptical of any methods attempting to predict the future of such complex systems. Any model assessment will be dependent on the assumption that the future will reflect the past. That the underlying statistics remain unchanged. If there is one constant in the world, it’s change. It’s not a question of whether you will be surprised in the future, it’s a question of when. Therefore instead of focusing on prediction, it seems much wiser to focus on awareness and rapid response. The important work you have done with Ushahidi fits exactly that paradigm and the results are clear. Unfortunately there are many that still believe prediction is the answer. With enough data and massive compute power, many think we will eventually develop the capability to predict the future state of such systems. I wish them much luck. In the meantime, the work to develop better awareness capabilities and networks that can rapidly adapt to changing environments must continue.

    • Many thanks for your comment, Chris. This does bring up another issue I’ve been thinking about: forecasting vs nowcasting. The latter is increasingly possible thanks to crowdsourcing methods/technologies. I imagine nowcasting approaches are far cheaper and in many ways perhaps more accurate. Sure, this is not the same as forecasting months in advance, but based on my experience in the field of conflict early warning, I can say that even if warnings are issued months/years in advance, this has rarely–very rarely–translated into preventive action.

  2. Oh this is such an interesting subject!

    It’s interesting that you mention Virginia Tech’s model. Several years ago I theorized about an agent-based model that divided everyone into what I simply termed ‘the 137 personality types.’ These personality types were based on some archetypal personalities I’d observed over and over through the course of my life and then the realization that mixing of these personalities created a tree and there were a number of bins (137… which is an estimate) that pretty much everyone fit into.

    I theorized that since the limbic brain is what controls the majority of our gut level decision-making processes that this type of model would outperform other types of models as the statistical sample became larger and the events you were talking about involved more people. (The ideal proving ground for this, I thought at the time, was the foreign exchange market.)

    I also theorized that there would be an upper limit to what you could predict based on complexity and other factors. You could make the model better by putting more and more data in, but past a certain point your diminishing returns would become exponential and you would run up against a series of asymptotes. (The asymptotes are another entire discussion, but I see at least one as being somewhat analogous to the uncertainty principal of not being able to know both a particle’s position and momentum.)

    Indeed statistical arbitrage is not the answer- sheer statistical arbitrage without context is incomplete at best. But on the other side of that… we are terribly predictable beings and we operate based on some pretty straightforward principals.

    I think the way they’ve tried to develop agent-based models is interesting, but limited to people who make rational decisions, which as far as I can tell isn’t any of us really – politicians probably get the closest – but my observation is that we only make some portion of our decisions based on logic thought and reason. And as your statistical sampling becomes larger you approach a different irrational – for lack of a better term – “limbic person” as the average, which I think is better characterized by a personality type making an emotional decision.

    However, with most tough questions – historically anyway – I’ve noticed that the best answer is usually somewhere in the middle. So the best model is probably a combination of context (Patrick, what I think you referred to in the Senturion model as the kind of ‘well duh!’ response,) complexity (the extremely complex web of relations, interactions, etc.), a bit of randomness (i.e. ‘that person was hit by a bus and won’t be coming to work today… or ever… and so no, they can’t sign that peace accord after all’) and sheer statistical arbitrage.

    A really good smart agent type model is probably in our future, but what interests me more than that is the crowdsourced “nowcasting.” I believe – much like you seem to believe … and Al Gore does believe – that we are at heart like frogs sitting in a cold pot slowly coming to a boil. If you give us warning, the gut-level decision making part of our brain doesn’t react because we don’t see or sense the danger… but when it becomes real, as in the case of a natural disaster, man do we turn-on fast. We donate, and volunteer, and you get all degree of impassioned response.

    This, I think, is a good place to concentrate our creative energy for now.

  3. Pingback: Two blog posts of possible relevance

  4. Pingback: Detecting Emerging Conflicts with Web Mining and Crisis Mapping | iRevolution

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s