December 7, 2012: The U.S. Army intelligence bureaucracy is in big trouble because they developed a data mining and analysis system that turned out to be more expensive and less effective than commercial products. A recent government investigation reports the problems in great detail. Over the last few years the U.S. Army has made a major effort to upgrade its intelligence tools to more accurately predict where roadside bombs and enemy leaders in Afghanistan are, as well as the location of enemy weapon storage sites, smuggling routes, and bomb making workshops. This was done using predictive analysis, which collects huge quantities of information and uses data mining and other tools to analyze it for patterns, which reveal things the enemy is trying to hide.

The army developed a new software system called DCGS (Distributed Common Ground System) to do this. DCGS was cobbled together on the fly, in the midst of a war and has not aged well. Recent investigations, in response to growing complaints from the troops, found that the army refused to recognize the problems with DCGS or get them fixed, or allow cheaper and more capable software to be used instead of DCGS.

For the last two years complaints from users and maintainers of DCGS have been getting louder (as in more politicians receiving emails about it). Some of the troops have been asking for more robust, powerful, and easier to use commercial data mining and predictive analysis software. The army complained that these commercial systems were expensive and required a lot of effort and money to integrate into DCGS. The troops insisted that this was not so and that commercial products like Palantir would save lives. Army bean counters insisted that it was probably only a few dozen lives, at most, and the additional money needed has to be taken from somewhere else, which might also cost lives in combat. But SOCOM and other organizations point out that they have been able to sneak Palantir into service in some areas and that arrangement outperformed DCGS.

The DCGS controversy also involves professional pride, as the army techs and managers have spent years building DCGS and are confident they can match any commercial products and do it cheaper. But that is rarely the case, as the army simply can't hire the best software engineers and project managers. When it comes to complex software systems, things go better if you keep an eye on the commercial side. If there is something there that does what you need done and does it faster, better, and more reliably it's worth paying the commercial price.

So far, when DCGS works, which is some of the time, the enemy finds this sort of thing very annoying. A sniper or smart bomb is something an Islamic terrorist can understand. Well, OK, the smart bombs smack of magic but these intel tools are incomprehensible to most everyone. Yet everyone in the United States is touched by these tools, every day. Fortunately snipers and smart bombs are not involved.

Thus it came to be that some of the most effective intelligence tools used in combat today (data mining and predictive analysis) were invented a century ago as part of the development of junk mail. Who knew? Now these tools predict what the enemy is going to do. For decades the statistical tools used to determine who to send junk mail to (so the sender would make a profit) were not much use to the military. Then came cheaper and more powerful computers and the development of data mining and analysis tools. This made a big difference because the more data you have to work with, the easier it is to predict things. This has been known for over a century.

By 2008, with thousand dollar laptop computers equipped with hundred gigabyte (or more) hard drives, you could put large amounts of data in one place, do the calculations, and make accurate predictions. This wasn't possible 40 years ago, when a 75 megabyte hard drive cost $45,000 and the computer doing the calculations cost even more than that. You also didn't have digital photography (more data you can store for analysis) or a lot of data, in general, stored electronically. It's all different today. That thousand gigabyte hard drive (holding over 10,000 times more data than the $45,000 drive of yore) cost less than a hundred bucks. The laptop running the analysis software would have qualified as a supercomputer a decade ago. Back then there were theories of how data analysis could predict things. Now all those theories are being put to the test and many have worked.

In the last six years intel analysts have realized how powerful their tools are. And for those who studied math, statistics, or business in college they know the power of data mining because it has become a very popular business tool. In places like Iraq and Afghanistan, lots of data is being collected all the time. It was some local data mining that led to the capture of Saddam Hussein, the death of al Qaeda-in-Iraq leader Zarqawi, and al Qaeda head Osama bin Laden. Over a hundred senior (team leader and up) al Qaeda terrorists have been killed or captured in Iraq using these techniques. The same thing is happening now in Afghanistan, although the introduction of DCGS has created a lot of complaints because, although DCGS had access to a lot more data, it often broke down or was too slow. Many units already had database and analysis systems they had put together (usually from commercial products) and found themselves using these "inferior" systems to get the job done when DCGS stumbled. This encouraged the DCGS developers to fix bugs more quickly but DCGS was still inferior, in many respects, to similar commercial software. Having spent nearly $3 billion on DCGS, the army was reluctant to admit defeat and spend a lot more on commercial software to replace parts of DCGS.

Meanwhile, the basic tasks DCGS must perform are not that complicated. Data mining is basically simple in concept. In any large body of data you will find patterns. Even if the bad guys are trying to avoid establishing patterns, they will anyway. It's human nature and only the most attentive pros can avoid this trap. Some trends are more reliable than others but any trend at all can be useful in combat. The predictive analysis carried out with data mining and other analytic tools has saved the lives of thousands of U.S. troops by giving them warning of where roadside bombs and ambushes are likely to be or where the bad guys are hiding out. Similarly, when data was taken off the site of a terrorist leader's death, it often consisted only of names, addresses, and other tidbits. But with the vast databases of names, addresses, and such already available, typing in each item began to generate additional information, within minutes. That's why, within hours, the trove of data can generate dozens of raids and even more leads. The enemy tried to adapt to all this and did to a certain extent. But the predictive analysis moves faster than the opposition can change and adapt. The only effective defense is a new enemy strategy, one that's a break with past practices. This sort of thing is very rare and not easily done. Even so, the predictive analysis eventually sorts it all out.

Speed has always been an advantage in combat but, until recently, rarely something intelligence analysis was noted for. This is no longer the case, at least as long as DCGS does not crash. Predictive analysis is something the troops depend on, not only for tips on what to avoid but for names and places to go after.

Israeli intelligence taught the Americans how to look for terrorist organizations and identify key leaders and technical specialists. These people became the key targets and that tactic enabled Israel to defeat the Palestinian terror campaign eight years ago. But this was done with old-fashioned police work and a network of informers inside Palestinian communities. The new computerized systems move data collection and analysis into the 21st century, using technology and concepts that many police departments are using to good effect. But being able to speak to the system, and have it understand what you are looking for, raises the intel game to a whole new level.