Drowning in Information: NSA Revelations From 262 Spy Documents

By the first half of 2004, the National Security Agency was drowning in information. It had amassed 85 billion phone and online call records and cut the ribbon on a new hacking center in Hawaii — but it was woefully short on linguists who could make sense of captured communications and lacked enough network analysts to effectively monitor all the systems it had hacked.

The signals intelligence collected by the agency was being used for critically important decisions even as NSA struggled to understand it. Some bombs in Iraq were being targeted based entirely on signals intelligence, a senior NSA official told staff at the time — with decisions being made in a matter of “minutes” with “less and less review.”

Information overload is just one of several themes running through 262 articles from the NSA’s internal news site, SIDtoday, which The Intercept is now releasing after careful review. The documents also detailed an incident in which the Reagan administration appears to have leaked classified intelligence to the press for political purposes, described in an accompanying article by reporter Jon Schwarz.

SIDtoday articles published today also describe how the NSA trained FBI agents, enabled U.S. intervention in Latin America, and, with the help of a gifted analyst at the Defense Intelligence Agency, learned the value of simply reading information that was already public. One document even suggests that NSA personnel routinely got dangerously chatty at restaurants near headquarters. These stories and more are described in the highlights reel below. The NSA declined to comment.

Dropping Bombs in Iraq “With Less and Less Review”

A top NSA official disclosed in a January 2004 SIDtoday column that U.S. forces were “dropping bombs” based entirely on signals intelligence, the type of intelligence collected by the agency. He then implied that the American officers involved risked prosecution for war crimes.

Charles Berlin, chief of staff in the Signals Intelligence Directorate, recounted an anecdote about a former commander of his who, in one session in the winter of 1995-96, personally reviewed more than 100 possible airstrike targets in the Balkans. The commander’s motivation, Berlin said, was to protect his underlings from being prosecuted for war crimes, and his actions “really brought home the concepts of responsibility and accountability.”

“For us today this lesson is especially important,” he added. “The planning cycle for dropping a bomb has compressed from a day to minutes and the criterion for the aiming point has less and less review.”

“As many of you know, our forces in Iraq are dropping bombs on the strength of SIGINT alone. We are proud of their confidence in us, but have you ever considered the enormous risk the commanders are assuming in this regard? Are you ready to share that risk?”

Inside the NSA’s Call-Logging Machine

Among the ways the NSA identified potential terrorists was through a practice known as “information chaining,” which uses communications metadata to draw a social graph. And there’s no question the agency had lots of metadata: As of 2004, the NSA had amassed a database of more than 85 billion metadata records related to phone calls, billing, and online calls — and was adding 125 million records a day, according to a January 2004 SIDtoday article titled “The Rewards of Metadata.”

The database, known as FASCIA II, would at some unspecified point in the future begin processing 205 million records a day and storing 10 years of data, the article added. One of the world’s largest Oracle databases at the time, FASCIA II held metadata records from telephone calls, wireless calls, billing, the use of media over the internet, and high-powered cordless phones, with plans to add email metadata in the future.

The article explained that metadata is used by the agency in the process of “information chaining,” in which analysts spy on relationships between people. It further claimed that two senior al Qaeda operatives had been captured with the help of such techniques. A March 2004 SIDtoday article said a chaining tool called MAINWAY helped a counterterrorism analyst uncover six new “terrorist-related numbers.”

Short on Linguists, NSA Struggled to Understand Targets

It’s one thing to collect phone calls, email messages, and other signals intelligence. It’s quite another to make sense of it. Several SIDtoday articles from the first half of 2004 made clear that the NSA was falling far short in its attempts to process communications conducted in languages other than English.

Only half of the agency’s more than 2,300 “language missions” worldwide had qualified personnel, according to a June 2004 SIDtoday article by an NSA “senior language authority.” The author declared that “this shortcoming must be rectified.” An NSA report to an oversight council, quoted in the article, said that the lack of qualified language analysts was particularly acute in the “Global War on Terrorism.”

Exacerbating the situation was the fact that captured communications require a high level of linguistic proficiency to understand. “The cryptologic language analyst must be able to read and listen ‘between the lines’ to unformatted, unpredictable discourse,” as the article put it. Only a quarter of military cryptologic linguists, who formed the vast majority of the workforce, could work at this level, known as “level 3” proficiency, while barely half of the civilian cryptologic linguists could, according to a follow-up SIDtoday article. The military’s language training institute offered “virtually no existing curriculum” above level 2.

NSA’s plan to address the problem included reforms to the training institute and on-site instruction to bring existing linguists up to higher levels. The agency planned to invest about $80 million per year in training over five years. Other efforts included an internal online language training tool, an evaluation of redundant Arabic machine translation projects underway in various government agencies, and the formation of a language technology team within the NSA.

How the NSA Over-Hacked

Sometimes metadata isn’t enough and the NSA decides it needs to compromise targets’ computers to collect much more data. The first half of 2004 saw a ramp-up of NSA’s hacking capabilities. In March, SIDtoday reported, the agency’s elite hacking team Tailored Access Operations approved Kunia Regional Security Operations Center in Hawaii — the same facility where Edward Snowden later worked — as the first NSA field office to conduct “advanced” Computer Network Exploitation. Other facilities conduct the first stage of hacking, “target mapping,” but the Kunia facility began doing “vulnerability scanning” all the way through to “sustained SIGINT collection.”

Another March SIDtoday article said that an advanced network analysis division used to help “exploit targets of interest” had “played an instrumental part” in capturing alleged al Qaeda operative Husam al-Yemeni, had developed a “more complete understanding of the Pakistani Army Defense Network (ADN) infrastructure,” and had assisted with the hacking of “an important digital network associated” with the leader of Venezuela at the time, referred to erroneously as “Victor Chavez.”

The NSA was so successful at hacking networks that the agency was overwhelmed with information. “We simply do not have enough network analysts to effectively monitor these targeted networks,” an NSA division chief wrote in an April 2004 SIDtoday article. To solve the problem, the agency began prototyping an automated monitoring system.

“Outstanding” Bookworm Spy Doesn’t Need to Really Spy

Even as the NSA made enormous efforts to collect vast quantities of private communications, a lone SIDtoday article extolled the value of publicly available data. The piece, from May 2004, gushed about a Defense Intelligence Agency analyst who dug up leads by poring over Russian material that was “open source.” The DIA bookworm searched in newspapers, government documents, and “obscure websites” for information that aided the NSA in collecting intelligence, including names, telephone numbers, and addresses. The article, co-authored by an NSA director with responsibility for Russia, praised the analyst’s “outstanding language and research skills.” It turned out that “critical lead information” on Russian underground facilities, including a mysterious and widely discussed site at Yamantau Mountain in the Urals, was “often only available in open source literature, such as the Internet.”

How the NSA Secures — and Routinely Puts at Risk — Sensitive Information

Knowing how much intelligence value could be reaped from openly circulated information, the NSA worked to encourage discretion among members of its workforce. NSA employees practiced poor operational security on a “monthly” basis by disclosing too much information in restaurants and other public settings near the agency’s Fort Meade headquarters, an agency security manager indicated in a tutorial on operational security that ran in SIDtoday in April 2004.

The article used a hypothetical scenario to explain why operational security, or OPSEC, was important for everyone. The author, OPSEC manager for the NSA’s Signals Intelligence Directorate, wrote: “You’re at a luncheon at a local restaurant to bid farewell to Sue, a co-worker who is moving on to a new office.” Your boss makes a toast to Sue, describing her contributions against organized crime and offering various details of her work. Sue then gives a toast thanking some of the gathered individuals.

“Sound familiar?” the OPSEC manager asked. “Then you’ve witnessed (or perhaps participated in) a demonstration of poor OPSEC. … Have you ever stopped to consider what your unclassified public discussions might be giving away? Take the scenario, for instance. This is a scene that is played out monthly in the Fort Meade area.” The article went on to list the pieces of information that an adversary, who could have been listening in from a nearby table, would have learned.

OPSEC turned out to be a recurring theme for SIDtoday — OPSEC training is, after all, mandatory for all NSA personnel. A January 2004 article, written by the author of the April 2004 piece, listed some tips to help personnel to apply OPSEC to their day-to-day activities: Identify your critical information, analyze the threat, identify vulnerabilities, assess risk, and apply countermeasures.

NSA employees aren’t the only ones trained to practice good OPSEC. A March 2005 article reported that the leaders of Venezuela and Cuba practiced OPSEC successfully. President Bush considered Venezuelan President Hugo Chavez a “threat to democracy in the region and a threat to U.S. interests in particular.” But “from a SIGINT perspective, Venezuela poses a particularly difficult challenge. With Castro as his mentor, Chavez has learned the importance of communications security and has made sure that his subordinates understand this as well.”

Law & Order & the NSA

Various 2004 SIDtoday articles highlight the NSA’s behind-the-scenes work on behalf of federal law enforcement.

One detailed a two-week training course on “intelligence reporting” given by NSA staff to FBI officers working on terrorism cases. The course, which had a component dubbed “SIGINT Reporting 101,” aimed to provide “insight into the complexity and difficulty of our business” and to dispel “Hollywood myths about the NSA.”

Another SIDtoday article showed how the U.S. Coast Guard was able to interdict a boat carrying 3.2 metric tons of cocaine thanks to the NSA’s monitoring of VHF radio signals, which carried voice communications of narcotraffickers. An official Coast Guard history of the incident elides the NSA’s role. The same SIDtoday article also disclosed that the Colombian air force carried out a strike against a suspected trafficker aircraft after a tip-off from the NSA.

NSA vs. FARC

Colombian guerrillas holding American hostages evaded massive NSA surveillance, according to a February 2004 SIDtoday article.

One year after three American contractors, who had been on a surveillance mission for the U.S. military, were captured by the Revolutionary Armed Forces of Colombia, a Marxist guerilla group, the U.S. “has not been able to determine with high confidence the exact location and status of the hostages,” wrote an NSA account manager for the military’s Southern Command. This despite “hundreds” of U.S. government personnel having worked to gain their release. U.S. efforts were stymied when FARC’s leadership ordered that personnel cease mentioning hostage operations directly in their communications; the best the NSA could achieve at the time of the SIDtoday article was to monitor calls between two radio operators, “Paula and Adriana,” who in turn were connected to the FARC leaders “we strongly suspect are linked to the hostages.”

The author of the SIDtoday article added that the agency continued to try and get a fix on the location of the hostages. Yet their captors eluded the Americans for another four years. The three Americans were freed by Colombian commandos in July 2008.

A March 2004 SIDtoday article noted a success against FARC, bragging that the arrest of FARC financial leader Anayibe Rojas Valderrama, known as “Sonya,” and a number of her associates a month earlier “resulted from years of monitoring. … Accurate geolocational data as to where she was and when, allowed a vetted Colombian team to capture them by surprise and without any loss of life.” Valderrama was extradited to the United States where she was tried and convicted on drug trafficking charges in 2007.

Internal NSA Criticism of Political Groups and the News Media

A national intelligence officer gave a top-secret “issue seminar” to NSA staff on the question of “where political action fades into terrorism,” according to a seminar announcement published in June 2004. The announcement suggested that the line between “legitimate political activity” and “activity that is the precursor to, or supportive of, terrorism” is fuzzy. The course used the Vienna-based organization Anti-Imperialist Camp as a case study, describing it as “ostensibly a political organization” but noting that “its many ties to terrorist organizations — and its attempts to collaborate with Muslim extremists — raise questions about where political action fades into terrorism.” No further details were given to substantiate the alleged ties; the group’s website remains online. A spokesperson for the group, Wilhelm Langthaler, told The Intercept that the group was targeted for such accusations for political reasons, including its opposition to the war in Iraq and “our public support for the resistance against occupation which we have compared with the antifascist resistance against German occupation.”

Another seminar announcement said the news media helped stymie U.S. intelligence collection. “A day hasn’t gone by that our adversaries haven’t picked up a newspaper or gone on the Internet to learn something new about how the US intelligence gathering system operates and what its capabilities or limitations are,” the course overview explained. “And in response, a day hasn’t gone by that our adversaries haven’t modified their operations and activities to avoid being detected and collected against by the US intelligence gathering system.”

NSA’s Role in the Failed Iran Hostage Rescue Attempt

In an anecdote about signals intelligence during the 1980 Iranian hostage rescue mission, a SIGINT staffer recalled the night of April 24 of that year, when he was told he was monitoring the ongoing “Operation Ricebowl.” In a May 2004 SIDtoday article, the staffer wrote: “We knew the parameters of the Iranian Air Defense system because it was U.S. equipment and installed by U.S. contractors while the Shah of Iran was still in power. We knew exactly where the gaps in coverage were and we exploited it during the rescue attempt.” The author went on to describe his shock the next morning when he saw on TV news at home that the mission had ended with a disastrous helicopter crash.

Top photo: American soldiers from the 1st Cavalry Division 2nd Battalion 7th Cavalry run through a smoke screen as they try to avoid sniper fire during an offensive operation on Aug. 16, 2004, in Najaf, Iraq.