A new government report reveals that federal agencies have undertaken 199 data-mining efforts, 131 of which are already operational. A surprising number resemble clones of the controversial Total Information Awareness project, which was intended to peruse exabytes of data on Americans assembled from every source possible as a means to snare terrorists.
Some of the highlights of the General Accounting Office report, which represents nearly a year of investigation:
The Department of Homeland Security is creating an Incident Data Mart designed to assemble data from every state, local, and federal police agency and spot possible terrorist activities. The data mart will sort through logs of incidents, defined as any "event involving a law enforcement or government agency for which a log was created, e.g., traffic ticket, drug arrest, or firearm possession."
Another data mart is across town at the FBI, which has compiled information from its own files, those of other federal agencies, and public data sources such as LexisNexis and court records. The purpose is "to determine unlawful entry" into the United States of potential terrorists.
Four projects at the Defense Intelligence Agency seem to be the most far-reaching of the lot. Their purpose is to mine "data from the intelligence community and Internet searches to identify foreign terrorists or U.S. citizens connected to foreign terrorism activities."
Even the Department of Education has a so-called Project Strikeback. That's designed to compare names in the department's databases with records supplied by the FBI and search for "anomalies" indicating "terrorist activities."
It's worth noting that government data mining can be very useful.
It's starting to seem like the government simply is unable to regulate itself.
But what should be done about government data mining that raises legitimate privacy concerns?
Current federal data mining efforts, which accelerated dramatically after the Sept. 11, 2001, terrorist strikes, are taking place with near-zero oversight from Congress or the public. The General Accounting Office's report is the first time some of these data-mining projects have been named publicly.
Last week, the American Civil Liberties Union and other civil liberties groups warned that data-mining is "creating a situation where Americans' every action, movement, and communication is likely to be recorded and stored in the memory of some computer database."
The usual solution would be to try to fix the problem by once again enacting more regulations. Congress could restrict controversial data-mining efforts. Information considered especially private, such as medical data, could receive additional protection. Americans could have a right to fix errors in government databases about them, and audit trails could track misuse by federal employees.
But it's starting to seem like the government simply is unable to regulate itself. After Congress shut down Total Information Awareness, its progeny popped up elsewhere. The 1974 Privacy Act slapped some limits on databases, but federal agencies found ways to bypass them. In the 2004 election, the McCain-Feingold campaign finance law merely prompted both major parties to invent new ways to inject money into politics.
That reality has convinced one of Washington's most ardent privacy proponents that incremental approaches simply won't work--and it's now time, in effect, to reformat the federal government's hard drive.
"It would be nice to catch fraud and waste" through data-mining, Rep. Ron Paul, R-Texas, told me Friday. "But I always argue that this has been going on for years. Conservatives use the argument too often: 'Let's just weed out the fraud and the waste.' It doesn't work. Government is by nature inefficient and harmful when it gets involved in these programs."
Paul asserts that the government shouldn't be running so many programs in the first place, including those related to education and health care.
"I think conservatives tend to lean in this direction: 'If we only monitor the programs, we can make them efficient enough so we can tolerate them,'" Paul said. "That's the question that needs to be challenged."
Declan McCullagh is CNET News.com's chief political correspondent. He spent more than a decade in Washington, D.C., chronicling the busy intersection between technology and politics. Previously, he was the Washington bureau chief for Wired News, and a reporter for Time.com, Time magazine and HotWired. McCullagh has taught journalism at American University and been an adjunct professor at Case Western University.