October 22, 2009: The CIA is developing the capability to monitor and analyze Twitter traffic, in addition to their existing ability to monitor email, message board and blog data. The Department of Defense and the CIA and other intelligence agencies are also adopting the "mashup" techniques used to quickly and easily combine visual and text data from many sources into a single screen full of more useful information. How else are you going to get anything useful out of millions of tweets and emails.

It's another example of the whole being more useful than the sum of the parts. The intelligence agencies have also become aware of how all this Internet data can give a faster, and more accurate, reading of public opinion in foreign countries. Although the Internet users are not a representative sample of the entire population, that can be adjusted for. Moreover, the Internet users are among the most active people in a society.

Mashups were popularized with the appearance of web apps like Google Maps and Google Earth, and the ability to overlay other web data on the maps and terrain photos. Before this, mashups were restricted to hand crafted business applications. Some of these used maps, but others were simply clever (and highly useful) arrangements of financial data (in lists and charts) to support financial operations. The most famous example of this was the "Bloomberg Box", which first appeared in the 1980s, and made financial analysis and trading much more efficient. This was preceded by MIS (Management Information Systems), which sought to combine the growing amount of computerized business data into a format that managers could more easily comprehend.

The military and intelligence agencies are using the mashup to rapidly assemble data for analysis and planning, as well as sorting out intelligence data. The military wants to give commanders the ability to quickly collect all available data (locally and worldwide) so that battle plans can be put together and executed before the enemy can realize, or react to the changing situation. Speed is a critical factor in planning, and lack of information was always a major impediment. As with the business world, the military now has enormous quantities of computerized data. The mashup techniques enable all this material to be quickly accessed and assembled in useful ways. But other mashups simply assemble data on opinions from many different Internet sources, and provide insights into what is really going on, or is about to happen.

Since September 11, 2001, intel analysts have realized how powerful these tools are. And for those who studied math, statistics or business in college, they know the power of data mining, because it has become a very popular business tool. In places like Iraq and Afghanistan, lots of data is being collected all the time. It was data mining that led to the capture of Saddam, and the death of al Qaeda leader Zarqawi, and most of his successors and associates.

Data mining is basically simple in concept. In any large body of data, you will find patterns. Even if the bad guys are trying to avoid establishing a pattern to their actions, they will anyway. It's human nature, and only the most attentive pros can avoid this trap. Some trends are more reliable than others, but any trend at all can be useful in combat. The predictive analysis carried out with data mining and other analytic tools has saved the lives of hundreds of U.S. troops, by giving them warning of where roadside bombs and ambushes are likely to be, or where the bad guys are hiding out. Similarly, when data was taken off the laptops of dead or captured terrorist leaders, it often consisted only of names, addresses and other tidbits. But with the vast databases of names, addresses and such already available, typing in each item began to generate additional information, within minutes. That's why, within hours, the trove of data generated dozens of raids, and even more leads.

Speed has always been an advantage in combat, but, until recently, rarely something intelligence analysis was noted for. No longer. Predictive analysis is something the troops depend on, not only to tips on what to avoid, but for names and places to go after. Adding this form of analysis to a mashup, it makes your next move more obvious, and further speeds up your decision making. The enemy has discovered that this speed is the most dangerous weapon the Americans possess. For intelligence agencies, it enables them to take many small, seemingly useless, bits of information, and rapidly tease out leads, and add them to the mashup.