November 18, 2009: The latest attempt by the FBI (the U.S. Federal Bureau of Investigation) to create a computer based case management system is nine months behind schedule and $18 million over budget. The system is now scheduled to come on line next September. The FBI has been working on this since the 1990s, and has had to cancel several failed efforts, costing the government billions of dollars. The current project began three years ago, and will cost over $400 million. While many in the FBI are eager users of computers, and early adopters of PC technology, there was an institutional resistance, in the organization, to widespread use of information technology. The FBI is currently in the midst of its third major effort to computerize the entire organization.

But that's not the only problem. Congress has regularly stymied the FBI from using technology that is commonly used by corporations, because of the fear that the FBI might abuse it. Case in point is data mining. Politicians tend to oppose this technique because access to too much information creates the possibility of the government oppressing people. Actually, politicians are well aware that data mining is more likely to uncover corrupt practices by politicians, than to cause any problems with the average voter.

The U.S. Congress has regularly refused to give the FBI money to expand the use of data mining in counter-terrorism efforts. American politicians are generally hostile to government use of data mining, despite the fact that it's a technique widely used, for decades, in business (marketing), law enforcement (catching criminals) and the military (finding the enemy). This last use has become much more sophisticated since the U.S. Department of Defense began pouring billions of dollars a year into finding ways to defeat IEDs (improvised explosive devices, usually roadside bombs). The effort to lower IED casualties has opened up all sorts of opportunities for technological innovation. No one harasses researchers for using data mining, or makes fun of building supercomputers with graphics processors (often the same ones found in video game consoles, making super-fast computers cheap enough to be used in a combat zone to make life saving predictions), when it saves troops from getting killed.

The data mining was initially used to figure out who the bomb making crews were, and where they operated from. Then, using math techniques first developed during World War II, the intel geeks began creating predictions about where IEDs were most likely to show up next. These predictive models get better as the quality of the information going into them improves. As more terrorists are captured and interrogated, and their computers and data is translated, the predictions become more accurate.

Using much more primitive computers, Germany employed data mining successfully in the 1970s, to find leftist, middle class terrorists who were operating with assistance from the East German secret police. The terrorists thought they were well concealed, but data mining can do wonders with the slightest pieces of information.

The FBI has had a difficult time making this point to Congress, mainly because some key legislators are ideologically opposed to data mining, and refuse to acknowledge the widespread success of the technique in civilians and military sectors. But the FBI persisted, and has successfully used data mining to detect criminal activity in many areas, including terrorism.