July 5, 2010: American computer firm IBM recently announced that it had developed a system (called Watson) that could be programmed to play the TV quiz game Jeopardy, and consistently win against the best human players. Watson uses the IBM Blue Gene supercomputer system to accomplish this. This has enormous military implications.

For intelligence professionals, this development is huge. For one thing, it means there is now a voice recognition system that can take a question, spoken by a human, understand it, and then have access to a data analysis system capable of quickly producing a spoken answer. This is important because, in the last decade, tools for capturing, storing and analyzing military intelligence data have grown more powerful and portable, as well as a lot cheaper. But to get at that data quickly and accurately, you have be an SQL (structured query language) ace. Actually, nearly everyone has been exposed to some SQL via using search engines like Google. But SQL itself is a much more powerful, and difficult to learn, beast. A system that allows intel officers, or commanders, to ask a series of spoken questions, and quickly get answers,  provides a powerful tool for effectively using the mountains of data collected. Yet, for the military, systems like Watson are seen as part of a trend.

This trend stretches back over three decades, to when the CIA first began planning to convert a lot of their top-secret data into computer databases, and more quickly access and analyze it. But it was the war in Iraq that really capitalized on this, and changed the way American military intelligence operations were conducted.

This came about because smaller combat units were now able to pass more information to the intelligence specialists, and these troops now had more powerful hardware and software to do something with the flood of new data. In the past, the brigade, division and higher level intelligence shops conducted a lot of their own intelligence gathering (and had their own aircraft and reconnaissance units for that purpose). In theory, they were supposed to be getting a lot of information from subordinate units. But that was a paperwork nightmare. The smaller units had lots of good intel, but they were out there getting shot at, and had a hard time filling out reports. Intelligence specialists were supposed to debrief patrols and troops involved in other combat operations. But, as always, there were shortages of manpower to collect the information and put it to use.

That began to change in Iraq, where a lot of the debriefing got automated, and even the smallest infantry patrol, could easily report their findings electronically. By capturing the data electronically at the lowest level, and building database tools to handle it, information did not expire (as battlefield information tends to quickly do).

With lots more fresh battlefield data available, it was now possible to do things that, only a few years earlier, were believed the province of much larger, and better funded, organizations. For example, there is data fusion. This is collecting intelligence from many sources, and sorting through it for useful patterns and items that, in seen in the right context, are very valuable. The first of these fusion centers were set up at the national level two decades ago. But during the course of the Iraq war, the concept moved down the food chain. Cheaper, and more powerful, computer hardware was able to use analysis software to speed the fusion process, even in the hands of a relatively inexperienced operator.

Some of the most effective intelligence tools used in combat today, data mining and predictive analysis, were invented a century ago along with the development of junk mail. Who knew? For decades, the statistical tools used to determine who to send junk mail to (so the sender would make a profit) were not much use to the military. Then came cheaper, and more powerful computers, and the development of data mining and analysis tools. This made a big difference, because the more data you have to work with, the easier it is to predict things. This has been known for centuries.

By 2008, with thousand dollar laptop computers equipped with hundred gigabyte (or more) hard drives, you could put large amounts of data in one place, do the calculations, and make accurate predictions. This wasn't possible thirty years ago, when a 75 megabyte hard drive cost $45,000, and the computer doing the calculations cost even more than that. You also didn't have digital photography (more data you can store for analysis), or a lot of data, in general, stored electronically. It's all different today. That hundred gigabyte hard drive (holding over a thousand times more data than the $45,000 one of yore) cost less than a hundred bucks.

In the last five years, intel analysts have realized how powerful their tools are. And for those who studied math, statistics or business in college, they know the power of data mining, because it has become a very popular business tool. In places like Iraq and Afghanistan, lots of data is being collected all the time. It was data mining that led to the capture of Saddam, and the death of Zarqawi. Actually, over a hundred senior (team leader and up) al Qaeda terrorists have been killed or captured in Iraq using these techniques.

Data mining is basically simple in concept. In any large body of data, you will find patterns. Even if the bad guys are trying to avoid establishing a pattern to their actions, they will anyway. It's human nature, and only the most attentive pros can avoid this trap. Some trends are more reliable than others, but any trend at all can be useful in combat. The predictive analysis carried out with data mining and other analytic tools has saved the lives of thousands of U.S. troops, by giving them warning of where roadside bombs and ambushes are likely to be, or where the bad guys are hiding out. Similarly, when data was taken off the site of a terrorist leader's death, it often consisted only of names, addresses and other tidbits. But with the vast databases of names, addresses and such already available, typing in each item began to generate additional information, within minutes. That's why, within hours, the trove of data can generate dozens of raids, and even more leads.

Speed has always been an advantage in combat, but, until recently, rarely something intelligence analysis was noted for. No longer. Predictive analysis is something the troops depend on, not only to tips on what to avoid, but for names and places to go after.

Israeli intelligence taught the Americans how to look for terrorist organizations, and identify key leaders and technical specialists. These people became the key targets, and that tactic enabled Israel to defeat the Palestinian terror campaign five years ago. But this was done with old-fashioned police work, and a network of informers inside Palestinian communities. The new computerized systems move data collection and analysis into the 21st century, using technology and concepts that many police departments are using to good effect. But being able to speak to the system, and have it understand what you are looking for, raises the intel game to a whole new level.