September 20, 2021: Over the last six years American Internet security researchers have come to realize that much of the data stolen by Chinese hackers could be obtained, without breaking any laws, from commercial data brokers. While “data privacy” is often a major political issue few of these prominent critics seemed to realize that most of this data was available from commercial firms long before the government began building their own computerized personnel databases, which were subsequently stolen by foreign hackers. Collecting such data for commercial purposes has been going on for over a century. American firms pioneered the business of selling to customers via mail-order catalogs. What revolutionized this business was the invention of the punch card and mechanical tabulating devices, a technology that lasted into the 1980s. These card -based databases were converted to digital files stored on hard drives and because of this the commercial data business grew enormously. With the Internet more public records were made available in digital form. At the same time credit card companies grew enormously and compiled enormous databases of customer transactions. Data analysis software appeared that was far more powerful than what could be done using the punch card data on tabulating machines. These older systems were far more capable than most people realized, but they used data that could be kept locked away, and unavailable online to users and hackers. Data brokers have been around since the age of punch cards and could provide impressive data analysis. The brokers sold the analysis capability and the data to their commercial customers and all the public knew was they were receiving a lot more mail solicitations (junk mail) that seemed to be customized to their personal buying habits. This was not new either, it’s what the mail order firms were doing since the late 19 ^th century. With computers and digital databases you can do it faster, more precisely and kept up-to-date for the customers of data brokers. No hacking was required, it was all done with data collected from government data (public records) and customer behavior still being collected by commercial firms. Those in the commercial data often sold custom datasets to government customers who were often amazed that all the data was commercial, none of it classified or stolen from the government. This became somewhat scary when the government data users realized that the government databases being stolen by foreign hackers could often be recreated legally by data brokers using their powerful data analysis software. Intel agencies were often unaware of such advances in data processing tech.

The hacker antics still monopolized the news. For example, in 2015 American SOCOM (Special Operations Command) personnel were dismayed when they all began receiving letters from OPM (Office of Personnel Management) confirming that unknown (but presumably Chinese) hackers had made off with their detailed (including background investigation material) personnel files. This includes fingerprints, details of family members and much more. The theft included all military personnel, including former members and the retired. Since the CIA recruits many of its field agents from former (often retired) SOCOM personnel, many key CIA people were now much less secret. It later turned out that commercial devices, like cell phones or exercise wristbands like Fitbit, were an even greater security problem. These devices were recognized as a security problem in mid-2018

Word quickly got around that this would not have happened if the United States had taken the same precautions that other Western nations, and even the CIA, take with the personnel records of key military and intelligence personnel. These precautions usually involve making it impossible to access those records via the Internet. OPM had not done that and instead relied on the belief that their Internet security measures were adequate.

The United States was already forced to admit that its Internet security efforts failed and that allowed critics within the Department of Defense to go public with the embarrassing reasons why. The main fault lies with poor leadership and that is seen in unwillingness to ensure that basic things, like making sure all systems are patched promptly when software publishers (especially for Operating Systems) make those patches available. Too many commanders let these patches accumulate because that’s an old habit in the military. Many commanders, and services (especially the air force) behave like their networks are patched and forget that all Department of Defense networks are connected, except for the ones deliberately kept off the Internet. These bad attitudes were worse in many civilian agencies, including, obviously, OPM. This eventually led to the realization that most government agencies were unaware what commercial data brokers could already do with public data.

All this is the result of a very embarrassing recent Internet based attack that led to the American accusations in 2015 that named China as the chief suspect in a hacker attack that made off with government databases containing personal information on nearly twenty million government employees (active and retired.) This included data collected for people applying for security clearances.

The Chinese connection appears to have been confirmed and a few American officials responsible for protecting networks were replaced, or even named. China has officially denied any involvement. Hackers can use the stolen information on 20 million Americans for various types of online larceny, or espionage or both. What was particularly worrisome, and made China look even more guilty, was the fact that none of the data had shown up on the Internet black market. Aside from Internet based fraud, the other major use of that data is espionage and trying to blackmail and turn current American intel personnel. Investigations into the Internet black market for data discovered that some of these crooks were selling legal data from data brokers that only looked like it was secret government data.

Meanwhile, even more serious problems were discovered that involved no hacking or illegal behavior. In mid-2018 the U.S. Department of Defense banned all personnel in “operational areas”, which were usually overseas combat zones, from using commercial devices with geolocation capability (GPS). This included cell phones and PSMs (Physiological Status Monitors) like Fitbit. What triggered this was the discovery that a social network for athletes called Strava had developed software enabling anyone to track users wearing a FitBit or other GPS enabled PSMs. Dedicated, especially professional, athletes joined Strava to exchange PSM information and that led to Strava developing features that enabled user locations worldwide to be tracked. Turns out that intelligence agencies had discovered Strava as well and reported that they could not only detect PSM users anywhere in the world but could often identify these users by name. Many intelligence and military personnel used their Fitbits while overseas, often on secret missions. From January to July 2018 the extent and implications of this became quite clear. The intel agencies quickly (and quietly) ordered their personnel overseas (and often at home as well) to stop using PSMs that made their data accessible to public networks, even ones that were not open to the public. These could be hacked. Now there is a market for “secure (encrypted) PSMs for military and intelligence personnel. Later came the discovery that commercial data brokers, using unclassified data and analysis systems could do the same work as the hackers and do it faster, at less cost and no risk of prosecution for espionage. At least not yet.