For three years, NSA promised a secret surveillance court that it was collecting "discrete" Internet communications about terrorists and spies, and not snooping on ordinary Americans. That turned out to be untrue, a newly declassified opinion revealed on Wednesday. In fact, the NSA was scooping up tens of thousands of Americans' emails, while assuring the court no such thing could possibly be happening.
The NSA made its guarantees because it was confident that the agency's systems could tell good guys from bad guys in the digital ether. They couldn't. And now, the myth of the National Security Agency's electronic omnipotence -- the myth that undergirds its massive power to pry into every aspect of our digital lives -- has taken another hit.
"Tens of thousands of wholly domestic communications" were inadvertently scooped up in NSA's digital dragnets, the court found, a tiny fraction of the total haul, but nonetheless a significant violation of the rules for handling Americans' private information.
"For the first time, the government has now advised the Court that the volume and nature of the information it has been collecting is fundamentally different from what the Court had been led to believe," wrote Judge John Bates of the Foreign Intelligence Surveillance Court in 2011. His opinion was declassified Wednesday amid increasing pressure for the Obama administration to reveal more about how and when the NSA monitors Americans' communications.
The problem was that the NSA was grabbing what the agency described as whole "transactions," or bundles or emails that were neither to, from, nor about the intended target. The NSA estimates it was collecting 56,000 communications per year in this manner for three years before officials discovered the problem and notified their overseers.
"That revelation fundamentally alters the Court's understanding of the scope of [NSA's] collection... and requires careful reexamination of many of the assessments and assumptions underlying its prior approvals," Bates wrote.
The court didn't find that the NSA was looking at Americans' emails without authorization or using that information in appropriately. The problem was with how the information was retained and apparently not protected from potential misuse.
In a briefing with reporters, a U.S. intelligence official, who declined to be identified, said the agency's systems had been taking a "screenshot" of the email inboxes of individuals who had been in touch with an original target.
Email communications travel the Internet as a single communication, the official said. "For technological reasons, the NSA was not capable of breaking those down, and still is not capable, of breaking those down into their individual components."
Bates also alluded to a problem with other NSA programs, including one that collects the metadata of Americans' phone records. The issue there concerned the terms NSA was using to search, or query, the massive database.
"Contrary to the government's repeated assurances, NSA had been routinely running queries of the metadata using querying terms that did not meet the required standard for querying," Bates wrote. "The Court concluded that this requirement had been 'so frequently and systematically violated that it can fairly be said that this critical element of the overall ... regime has never functioned effectively.'"
Elsewhere, Bates wrote, "The Court is troubled that the government's revelations ... mark the third instance in less than three years in which the government has disclosed a substantial misrepresentation regarding the scope of a major collection program."
It is technically impossible to avoid collecting U.S. persons' emails when the NSA is targeting those of foreigners. So, rather than halt the surveillance, the NSA changed its minimization procedures in a way that the court found acceptable.
The government developed the technical means to segregate data that is mostly likely to have involved U.S. persons, after it is collected. Data that cannot be segregated is subjected to limits on how it can be used and disseminated, according to another U.S. intelligence official, who also declined to be identified. And any information retained in this way must be purged from NSA's systems in two years. Some other U.S. person information, such as encrypted emails or metadata phone records, can be kept for up to five years before it must be discarded.
In what intelligence officials described as an attempt to be more transparent with Americans about how the NSA gathers intelligence, officials announced that they'd set up a Tumblr page for declassified documents, official statements, testimony, and other materials.
The declassified court opinion was not posted to the site when officials held a briefing with reporters at 3 P.M. Wednesday. Several journalists complained they were unable to ask detailed questions about the overcollection of emails since they hadn't' had a chance to read the court's lengthy opinion.
John Reed reports on the frontiers of cyber war and the latest in military technology for Killer Apps.