November 6, 2014: U.S. SOCOM (Special Operations Command) is again trying to establish the capability to quickly scour the Internet for publically available information (OSINT or Open Source Intelligence) on an area or people SOCOM operators will be working with (or against.) SOCOM has sought to obtain this capability in the past but has been thwarted by special interest groups accusing them, and other government organizations, of violating the privacy of Americans. This is politically explosive as the media loves to use this sort of thing to grab attention. Yet SOCOM is only trying to do what marketing firms, intelligence agencies and election consultants do regularly, and legally. SOCOM needs this capability because they are being asked to send their operators, on short notice, to lots more places and more frequently. On short notice, and for only a small group of operatives, SOCOM usually cannot get the information they need from the intel agencies. At the same time SOCOM knows they do not need classified data, which the intel organizations prefer to deal with. The open source data available on the Internet often gets the job done, if only they could get past political grandstanding, special interest groups and headline hungry media.

While the U.S. intelligence community long resisted recognizing the importance and usefulness of OSINT over the last few decades, the enthusiastic acceptance of Internet-based OSINT by so many individual military personnel and commercial information gatherers has led to growing official government acceptance of what many intelligence professionals now consider a crucial tool and one that can only grow in usefulness.

The Internet has made OSINT a really, really huge source of useful intelligence. It's not just the millions of gigabytes of information that is placed on the net but the even more voluminous masses of message board postings, blogs, emails, and IMs (instant messaging) that reveal what the culture is currently thinking. It was corporate intelligence practitioners who alerted the government intel people to the growing usefulness of Internet based data. Corporations have developed, over the last few decades, a keen interest in gathering intel on competitors, new markets, and all manner of things that might affect them. The Internet has made this a much more useful and affordable exercise, especially since corporations are less likely to break the law when gathering intel, or have access to the powerful legal tools available to government investigators and analysts.

For years corporate intel specialists were concerned that government agencies, especially the CIA, were not taking sufficient advantage of OSINT. Part of the problem was cultural. The intelligence agencies have always been proud of their special intel tools, like spy satellites, electronic listening stations, and spy networks. Most of these things are unique to government intelligence operations. People who use this stuff tend to look down on a bunch of geeks who simply troll the web. Even when the geeks keep coming up with valuable stuff, they don't get any respect. Of at least they didn’t for a long time. That began to change after September 11, 2001, when many intelligence specialists, who were reservists, were called to active duty. Many of these men and women worked in BI (Business Intelligence, sometimes called corporate spying) and brought with them a respectful attitude towards OSINT and spectacular (to the government intel people) ability to use it.

Before long many junior members of the intel agencies were using OSINT more frequently. Then it was pointed out that there was growing evidence that some foreign countries were exploiting OSINT (especially the Internet) more effectively than the United States. No foreign intel agency will admit to this, but there are indications that some nations are mining the Internet quite intensively and effectively. Data mining is a heavily used commercial tool that the U.S. intel agencies have used, but now they have adopted the corporate techniques of plowing through vast quantities of unclassified data and often finding gold.

An example of this official acceptance occurred in 2012, when the U.S. Army issued a manual, Army Techniques Publication 2-22.9, which detailed how to use open source (mainly searching the Internet) intelligence most effectively. This was the kind of OSINT troops had been using for over a decade. The publication of ATP 22.9 was a way for the senior army leadership to say, "message received and understood." ATP 22.9, despite all the useful tips it contains, won't go far in helping the many soldiers already using the Internet, but it will be useful in convincing their bosses that a lot of useful stuff can be obtained from the Internet.

The government and military intel community has the money and software chops to screen and analyze huge quantities of data on the Internet, both text and pictures. Despite all these resources the intel behemoths continue to get overtaken by civilian amateurs. A large factor in this was the appearance of Google Earth and other commercial satellite photo sources. This revolutionized military intelligence and the way news on military affairs is developed and spread. Case in point was details on the transformation of the Chinese armed forces and the activities of the North Korean military. Both China and North Korea have long been very secretive about military affairs. But the appearance of Google Earth (originally as Earth View) in 2005 changed everything. By putting so much satellite photography at the disposal of so many people, in such an easy- to- use fashion, unexpected discoveries were made.

People soon discovered that if they had a high-speed Internet connection, they could use Google Earth to find satellite photos of all sorts of interesting stuff. This was especially true of the "Forbidden Kingdoms" (China, Russia, North Korea, and a few others). While the CIA and the military has had access to satellite photos of these countries since the 1960s, little of it was shown to the public. Now that so many people can examine these lower resolution, civilian satellite images many have gone over vast stretches of the Forbidden Kingdoms and found things that were newsworthy and never reported before. Things like new military bases, test sites for new weapons, and the new weapons themselves. The open discussion of these findings, most of them already known to the large national intel agencies, brought forth insights and analysis that was often superior to what the much smaller number of professional analysts were capable of. Another example of “the wisdom of the crowd.”

Technically, the countries in question can request that Google not show these classified military facilities. But in making that request, they point out where the classified operation is. So far, a lot of this stuff is just there to find. And users find it. This is called "crowdsourcing" (where large numbers of people accomplish impressive feats of research or analysis because they can quickly mobilize and get to the task via the Internet). The U.S. military will not say that they appreciate the work done via crowdsourcing, but individual analysts and intelligence officials have made it known, unofficially, that crowdsourcing is another useful tool that unexpectedly came their way via the Internet.

While privacy advocates do not criticize individuals, or groups of people cooperating via the Internet, for finding these use bits of information, it is a different matter if a military or government organization does it. As a result SOCOM has a much more difficult time with the political, rather than the technical, aspects of OSINT.