American politicians are generally hostile to government use of data mining,
that technique is widely used in business (marketing), law enforcement
(catching criminals) and the military (finding the enemy). This last use has
become much more sophisticated since the U.S. Department of Defense began
pouring billions of dollars a year into finding ways to defeat IEDs (improvised
explosive devices, usually roadside bombs). Last year, these bombs accounted
for about half of U.S. casualties in Iraq, now that's up to 75 percent. The
effort to lower IED casualties has opened up all sorts of opportunities for
technological innovation. No one harasses researchers for using data mining, or
makes fun of building supercomputers with graphics processors (often the same
ones found in video game consoles). This last item makes super-fast computers
cheap enough to be used in a combat zone to make life saving predictions.
The data mining was
initially used to figure out who the bomb making crews were, and where they
operated from. Then, using math techniques first developed during World War II,
the intel geeks began creating predictions about where IEDs were most likely to
show up next. By then (2005), American forces had a highly skilled and capable
Night Shift that patrolled roads and looked for IEDs, and the people who placed
them, before the bombs could hurt anyone. The latest iterations of this
predictive systems operate in real time, and live feeds of the data can be sent
to combat units, or the Night Shift units looking to shut down or destroy bombs
before anyone gets hurt (except for the terrorists caught placing the bombs).
These predictive models
get better as the quality of the information going into them improves. As more
terrorists are captured and interrogated, and their computers and data is
translated, the predictions become more accurate. This perplexes the
terrorists, who know of these models. The terrorists with degrees in math
(there are a number of those) often have a good idea of how the models work,
but these less educated brethren are simply frightened of this "infidel magic."
That's good too, at least for us.