The International Studies Review just published a piece by Sean O’Brien entitled “Crisis Early Warning and Decision Support: Contemporary Approaches and Thoughts on Future Research.” Sean outlines the latest attempt by the US military (ie, DARPA) to develop a crisis forecasting tool. This time, the platform is called ICEWS for Integrated Crisis Early Warning System.
O’Brien gives a brief overview of recent efforts in this space including Bueno de Mesquita’s Policon and Senturion forecasting systems, which are said to be 90%+ accurate. That said, O’Brien notes that Mesquita himself acknowledges that “he is not exactly sure how to interpret this accuracy claim since most of the reported assessments he has were not explicit about how accuracy was measured.” Incidentally, I like how this rather important qualifier is buried at the bottom of a footnote.
Some of Policon’s/Senturion’s supposedly accurate predictions had to do with questions like:
- What policy is Egypt likely to adopt towards Israel?
- What is the Philippines likely to do about US bases?
Keep in mind that millions of dollars were spent on these sophisticated systems and yet I can’t help but think that paying some experts on Egypt and the Philippines a few thousand dollars would have more or less accomplished the same task. Apparently, Senturion accurately predicted the deteriorating disposition of Iraqis toward US forces. Really? Shocking, who would have expected Iraqi public opinion to shift? Yes, that was sarcasm.
O’Brien also references a forthcoming study by Ward, Greenhill and Bakke, which “delivers a serious blow to the predominant way in which most conflict models are evaluated using statistical significance.” These include the predictive models developed by Fearon and Latin (2003) and Collier and Hoeffler (2004). Ward et al. show that these models predict few if any civil war cases at a reasonable probability cut off of 50%. In fact, the Fearon and Latin model “does not even appear to generate a probability of greater than 30%.” In sum, Ward et al. conclude that we cannot correctly predict over 90% of the cases with which our models are concerned.
Many of the most interesting, policy-relevant theoretical questions are also the most complex, nonlinear, and highly context-dependent. They demand consideration of hundreds of massively interacting variables that are difficult to measure systematically and at a level of granularity consistent with the theory. In such cases it is at best impractical and at worst impossible to apply standard regression techniques within the context of a Large N study, short of invoking unreasonable, oversimplifying assumptions. This may in part account for contradictory findings in the literature relative to the validity of alternative theoretical claims.
So lets keep in mind that previous “breakthroughs” have since been largely discounted.
ICEWS phase one of three consisted of a competition between different groups to successfully predict events of interest (EoI) on a set of historical data. The most successful team was Lockheed Martin-Advanced Technology Laboratories (LM-ATL) in cooperation with a number of established scholars and industry partners. The team integrated and applied six different conflict modeling systems, including:
- Agent-based models drawn from Barry Silverman’s Factionism and Ian Lustick’s Political Science-Identity (PSI) computational modeling platforms. The latter is created with “agents representing population elements of various ethnic ⁄ political identities organized geographically and in authority structures designed to mirror the society being studied.”
- Logistic regression models developed by Phil Schrodt and Steve Shellman, which use “macro-structural and event data factors commonly analyzed in the academic literature.” Shellman’s approach uses a Bayesian statistics model.
- Geo-spatial network models built by Michael Ward, which uses “structural factors, event counts, and various types of spatial networks—trade ties, people flows, and ‘‘social similarity’’ profiles—that embody potential EOI co-dependencies between proximate countries.”
- “A final model was developed by aggregating the forecasts from the above mentioned models using Bayesian techniques.
I’m particularly interested in the use of Agent-based models (ABM) for conflict analysis. O’Brien references a very interesting project at Virginia Tech which I was unaware of:
Scholars at Virginia Tech have already developed a 100 million agent simulation that includes synthetic versions of many American citizens, and plan to expand to 300 million agents this year (Upson 2008). Each synthetic agent has as many as 163 variables describing age, ethnicity, socio-economic status, gender, and various attitudinal factors. The simulation is used to assess how different types of pandemics could spread across the United States under different scenarios.