Intelligence: Overwhelmed by Success and Paranoia


June 21, 2006: The current American efforts to "improve intelligence" continue to run into two seemingly intractable problems. First, agencies continue refusing to share data. The usual excuse, that such sharing would threaten sources, is wearing thin. The National Intelligence Director has asked for examples of such damage to sources, and has not received any. The fanatical dedication to the protection of sources is hard to overcome. A particularly valuable source is usually someone on the inside, who could be killed if the connection to American intelligence services was revealed. But as a practical matter, the vast majority of the "burned" (exposed, and often killed) sources were the result of a traitor within the intel or law enforcement agency. But logic has nothing to do with this. The institutional desire to protect sources is so strong that a major effort will be required to effect a change. The smart money is on things staying the way they are.

The other problem, the inability to analyze all the information collected, is more likely to be solved. This was a problem that arose during the Cold War, when spy satellites and electronic eavesdropping systems were developed, and quickly began to generate far more data than could be examined and analyzed. This is still a major problem. For the last two decades, there have been several major efforts to deal with the flood of data using computer systems. Huge databases and software that can scan large amounts of data quickly and extract the useful bits, were developed. Typically, only a few percent of the material collected (including a lot of written reports) are useful. It's really a needle in the haystack problem.

The current war on terror has, as wars tend to do, spurred development of more effective screening and analysis tools. This has put the spotlight back on the sharing problem. The more data you have, the better the screening software can work. This has also led to another problem. Some political groups have invoked fears of 1984 and "Big Brother" by calling the new techniques invasive and a threat to personal liberties. This makes politicians leery of getting behind any large scale automated data analysis systems. The 1984 angle is more hysteria than anything else. It comes up any time some pressure groups notice that the quantity of personal data has been growing at an enormous rate over the last few decades, and that the government, as well as commercial firms, can analyze it. This is much ado about nothing, but it makes for great headlines, and can easily be used to frighten people. The military gets around all this by calling the analysis systems something else, and keeping quiet about it.

But because of the institutional, technical and political problems, the intel agencies are still overwhelmed with data. The new software sifting systems are showing results, but only here and there. No big breakthroughs, and when these do occur, no one will be releasing press releases.


Article Archive

Intelligence: Current 2022 2021 2020 2019 2018 2017 2016 2015 2014 2013 2012 2011 2010 2009 2008 2007 2006 2005 2004 2003 2002 2001 2000 1999 



Help Keep Us Soaring

We need your help! Our subscription base has slowly been dwindling. We need your help in reversing that trend. We would like to add 20 new subscribers this month.

Each month we count on your subscriptions or contributions. You can support us in the following ways:

  1. Make sure you spread the word about us. Two ways to do that are to like us on Facebook and follow us on Twitter.
  2. Subscribe to our daily newsletter. We’ll send the news to your email box, and you don’t have to come to the site unless you want to read columns or see photos.
  3. You can contribute to the health of StrategyPage. A contribution is not a donation that you can deduct at tax time, but a form of crowdfunding. We store none of your information when you contribute..
Subscribe   Contribute   Close