28 January 2013
ATT Project Greenstar Secretly Spied Millions of Calls
Greenstar prefigures current ATT's once-secret participation in intercepting
vast telecommunications data for the National Security Agency. More:
EXPLODING THE PHONE
The Untold Story of the Teenagers and Outlaws Who Hacked Ma Bell
Grove Press New York
If there were no billing records for fraudulent calls, there was no way to
know how many fraudulent calls there were or how long they lasted. And that
meant AT&T was gazing into the abyss. Say the phone company catches some
college students with electronic boxes. Fantastic! But elation is soon replaced
by worry. Is that all of them? Or is that just the tip of the iceberg? Are
there another ten college students doing it? A hundred? Are there a thousand
fraudulent calls a year or are there a million?
Engineers hate stuff like this.
Bell Labs, filled to the brim with engineers, proposed a crash program to
build an electronic toll fraud surveillance system and deploy it throughout
the network. It would keep a watchful eye over the traffic flowing from coast
to coast, ever vigilant for suspicious calls -- not every call, mind you,
but a random sampling of a subset of them, enough to gather statistics. For
the first time Bell Labs -- and AT&T's senior management -- would have
useful data about the extent of the electronic toll fraud problem. Then they'd
be in a position to make billion-dollar decisions.
The project was approved; indeed, AT&T gave Bell Labs a blank check and
told them to get right to work. Tippy-top secret, the program had the coolest
of code names: Project Greenstar. Within Bell Labs Greenstar documents were
stamped with a star outlined in green ink to highlight their importance and
sensitivity. Perhaps as a joke, the project lead was given a military dress
uniform hat with a green general's star on it, an artifact that was passed
on from one team lead to the next over the years.
Greenstar development began in 1962 and the first operational unit was installed
at the end of 1964. Bill Caming, AT&T's corporate attorney for privacy
and fraud matters, became intimately familiar with the program. "We devised
six experimental units which we placed at representative cities," Caming
said. "Two were placed in Los Angeles because of not only activity in that
area, but also different signaling arrangements, and one was placed in Miami,
two were originally placed in New York, one shortly thereafter moving to
Newark, NJ, and one was placed in Detroit, and then about January 1967 moved
to St. Louis."
Ken Hopper, a longtime Bell Labs engineer involved in network security and
fraud detection, recalls that the Greenstar units were big, bulky machines.
"I heard the name 'yellow submarine' applied to one of them," he says. They
lived in locked rooms or behind fenced-in enclosures in telephone company
switching buildings. A single Greenstar unit would be connected to a hundred
outgoing long-distance trunk lines and could simultaneously monitor five
of them for fraud. The particular long-distance trunk lines being monitored
were selected at random as calls went out over them. At its core, Greenstar
looked for the presence of 2,600 Hz on a trunk line when it shouldn't be
there. It could detect both black box and blue box fraud, since both cases
were flagged by unusual 2,600 Hz signaling.
As Caming described it, "there were in each of these locations a hundred
trunks selected out of a large number, and the [ ... ] logic equipment would
select a call. There were five temporary scanners which would pick up a call
and look at it with this logic equipment and determine whether or not it
had the proper [ ... ] supervisory signals, whether, for example, there was
return answer supervision. When we have a call, we have a supervisory signal
that goes to and activates the billing equipment which usually we call return
answer supervision. That starts the billing process and legitimizes the call,
and if you find voice conversation without any return answer signal, and
that is what it was looking for, it is an indication, a strong indication,
of a possible black box that the caller called in; and if, for example, you
heard the tell-tale blue box tone [ ... ] this was a very strong indication
of illegality because that tone has no normal presence upon our network at
When Greenstar detected something unusual, it took an audacious next step:
it recorded the telephone call. With no warrant and with no warning to the
people on the line, suspicious calls were silently preserved on spinning
multitrack reel-to-reel magnetic tapes. If Greenstar judged it had found
a black box call it recorded for sixty to ninety seconds; if it stumbled
upon a blue box it recorded the entire telephone call. Separate tracks recorded
the voice, supervisory signals, and time stamps.
When the tapes filled up they were removed by two plant supervisors. "They
were the only two who had access from the local [telephone] company," Caming
says. Then they were sent via registered mail to New York City. There, at
the Greenstar analysis bureau, specially trained operators -- "long-term
chief operators who had great loyalty to the system [who] were screened for
being people of great trust," Ken Hopper says -- would listen to the tapes,
their ears alert for indications of fraud. The operators would determine
whether a particular call was illegal or was merely the result of an equipment
malfunction or "talk off" -- somebody whose voice just happened to hit 2,600
Hz and had caused a false alarm. When these operators were finished listening,
the tapes would be bulk erased and sent back for reuse.
"The greatest caution was exercised," Bill Caming recalls. "I was very concerned
about it. The equipment itself was fenced in within the central office so
that no one could get to it surreptitiously and extract anything of what
we were doing. We took every pain to preserve the sanctity of the recordings."
Project Greenstar went on for more than five and a half years. Between the
end of 1964 and May 1970, Greenstar randomly monitored some 33 million U.S.
long-distance phone calls, a number that was at once staggeringly large and
yet still an infinitesimally tiny fraction of the total number of long-distance
calls placed during those years. Of these 33 million calls, between 1.5 and
1.8 million were recorded and shipped to New York to be listened to by human
ears. "We had to have statistics," said Caming. Statistics they got: they
found "at least 25,000 cases of known illegality" and projected that in 1966
they had "on the order of 350,000 [fraudulent] calls nationwide."
"Boy, did it perk up some ears at 195 Broadway," says Hopper. It wasn't even
that 350,000 fraudulent calls was that big a number. Rather, it was the fact
that there was really nothing that could be . done about it, at least not
at once. "It was immediately recognized that if such fraud could be committed
with impunity, losses of staggering proportions would ensue," Caming said.
''At that time we recognized -- and we can say this more confidently in public
in retrospect -- that we had no immediate defense. This was a breakthrough
almost equivalent to the advent of gunpowder, where the hordes of Genghis
Khan faced problems of a new sort, or the advent of the cannon."
The initial plan with Greenstar was simple: Wait. Watch. Listen. Gather
statistics. Tell no one. Most important, don't do anything that would give
it away. "There was no prosecution during those first couple of years," Hopper
says. "It was so the bad guys would not be aware of the fact that they're
being measured." It was only later, Hopper says. that AT&T decided to
switch from measurement to prosecution. Even then! Hopper said, "The presence
of Greenstar would not be divulged and that evidence gathered to support
toll fraud prosecutions would be gathered by other means." Instead, Hopper
relates, Greenstar would be used to alert Bell security agents to possible
fraud. The security agents would then use other means, such as taps and
recordings, to get the evidence needed to convict. "Greenstar bird-dogging
it would not be brought out," says Hopper. "It was just simply a toll fraud
investigation brought about by unusual signaling and you would not talk about
the fact that there was a Greenstar device. That was the ground rule as I
understood it. Any court testimony that I ever gave, I never talked about
any of that." As another telephone company official put it, "If it ever were
necessary to reveal the existence of this equipment in order to prosecute
a toll fraud case, [AT&T] would simply decline to prosecute."
Bill Caming became AT&T's attorney for privacy and fraud matters in September
1965. Greenstar had been in operation for about a year when he was briefed
on it. His reaction was immediate: "Change the name. I don't even
know what it is, but it just sounds illegal. Change the name."
More innocent-sounding code names like "Dewdrop" and "Ducky" were apparently
unavailable, so AT&T and Bell Labs opted for something utilitarian and
unlikely to attract attention: Greenstar was rechristened "Toll Test Unit."
As the new legal guy at AT&T headquarters, Caming faced questions that
were both important and sensitive. Forget how it sounded, was Greenstar actually
illegal? And if it was, what should be done about it? Before joining AT&T
Caming had been a prosecutor at the Nuremberg war crimes trials after World
War II. He was highly regarded, considered by many to be a model of legal
rectitude. Was there any way he could see that the AT&T program was legit?
There was. He later stated under oath that there was "no question" Greenstar
was in fact legal under laws of the day -- a surprising conclusion for what
at first blush appears to be an astonishing overreach on the part of the
telephone company. There were two parts to Caming's reasoning. The first
had to do with the odd wording of the wiretap laws of the early 1960s; using
this wording Caming was able to thread a line of legal logic through the
eye of a very specific needle to conclude that the program was legal under
the law prior to 1968. The second part had to do with his position at American
Telephone and Telegraph. In 1968, when Congress was considering new wiretapping
legislation, Caming was in a position to help lawmakers draft the new law.
He made very sure that the new wiretap act didn't conflict with AT&T's
Caming even informed the attorneys at the Justice Department's Criminal Division
about Greenstar in 1966 and 1967, in connection with some prosecutions. "Now,
that does not say that they cleared it or gave me their imprimatur," he allowed.
But then, he added, "we did not feel we needed it."
Years later, the Congressional Research Service agreed with Caming regarding
the legality of the program -- to a degree. While not going so far as to
say there was "no question" that Greenstar was legal, it was concluded that
"It is not certain that the telephone company violated any federal laws by
the random monitoring of telephone conversations during the period from 1964
to 1970. This uncertainty exists because the Congressional intent [in the
law] is not clear, and case law has not clearly explained the permissible
scope of monitoring by the company."
This whole mess formed a challenging business conundrum for AT&T executives,
the sort of thing that would make for a good business school case study.
Put yourself in their shoes. You have made an incredibly expensive investment
in a product -- the telephone network -- that turns out to have some gaping
security holes in it. You have, as Bill Caming said, no immediate defense
against the problem. You finally have some statistics about how bad the problem
is. It's bad, but it's not terrible, unless it spreads, in which case it's
catastrophic. Replacing the network will take years and cost a billion dollars
or so. The Justice Department isn't sure there are any federal laws on the
books that actually apply. And every time you prosecute the fraudsters under
state laws, not only do you look bad in the newspapers -- witness the
Milwaukee Journal's 1963 front-page headline "Lonely Boy Devises Way
of Placing Free Long Distance Calls" -- but the resulting publicity makes
the problem worse.
AT&T played the best game it could with a bad hand. For now it would
quietly monitor the network, keeping a weather eye on the problem. When the
company found college kids playing with the network, investigators would
give them a stern talking -- to and confiscate their colored boxes. Execs
would start thinking about a slow, long-term upgrade to the network to eliminate
the underlying problem. And if opportunity knocked and they could help out
the feds with an organized crime prosecution -- and in the process set a
clear precedent for the applicability of the federal Fraud by Wire law --
well, that would be lovely.
That opportunity came knocking in 1965. As it turned out used a sledgehammer.
On May 5, 1969, the Supreme Court declined to hear their case. More than
three years after the FBI took a sledgehammer to Ken Hanna's door, the issue
was finally settled. If you were making illegal calls you had no right to
privacy. The phone company could tap your line and turn the recordings over
to law enforcement.
For the phone company, the victory was about much more than convicting Hanna
or Dubis. AT&T now had a case that had gone all the way to the Supreme
Court, one that proved, definitively, that 18 USC 1343 -- the Fraud by Wire
law that the Justice Department had believed wasn't relevant -- did apply
to blue boxes. Thanks to Hanna's failed appeal, the matter was now settled.
AT&T finally had an arrow in its quiver to use against the fraudsters.
Throughout all of this legal drama one mystery remains: how had the telephone
company found out about Hanna's or Dubis's blue box calls in the first place?
In the Hanna case, Miami telephone company security agent Jerry Doyle received
a telephone call from the Internal Audit and Security Group at AT&T
headquarters in New York asking him to investigate Hanna's telephone line
for a possible blue box. How did investigators in New York know that somebody
in Miami was making illegal calls? Hanna's attorneys asked Doyle this very
question but Doyle said he didn't know.
There was a one-word answer that nobody was giving: Greenstar. Hanna had
been caught up in AT&T's toll fraud surveillance network. Imagine what
would have happened if this had come out during Hanna's trial. After all,
the Hanna case took almost four years to resolve and went to the Supreme
Court based on tape recordings of each of his illegal calls. Think of the
legal circus that would have ensued if Hanna's defense attorneys had learned
that the telephone company had been randomly monitoring millions of telephone
calls nationwide and recording hundreds of thousands of them.
This added considerably to the stress of prosecuting Greenstar cases. AT&T
attorney Caming recalls, "That was the problem in the Hanna case! Fortunately,
defense counsel never probed too far as to what our original sources of
information were." With blue box prosecutions, he adds, "We were always on
pins and needles as to what might spill over into the public press."
Fortunately for AT&T in the Hanna and Bubis cases their luck held. And
although Caming wasn't a gambler or a bookmaker, he knew a thing or two about
luck. In particular, he knew it didn't last forever.
At that point, the phone company billing records show something anomalous:
here's a call to a number, 555-1212, that should never look like it answered
and yet it does. The phone company doesn't like anomalies in its network,
not so much because they think somebody might be messing with them, but just
because anomalies probably mean that something is broken somewhere and needs
"I knew that was an irregularity," Acker says. "My fear was, you know, if
this registers on your tape" -- Acker knew the phone company in those days
used paper tape for billing records -- "they'll be able to tell that [the
call] answered, and they know it's not supposed to." Acker's fears were right
on the money. The phone company was indeed using computer-generated reports
of supervision irregularities to spot blue boxes. Along with Greenstar, these
reports were a primary tool the Bell System used to detect such fraud and,
due to Greenstar's secrecy, were among the most effective for prosecution.
Acker's surprise caller was a security agent from his telephone company,
New York Telephone. The agent had already talked to Acker's friend John,
likely because of 555-1212 supervision anomalies. But the reason the agent
wanted to talk to Acker was more concrete. John had ratted out Acker to the
"He spilled his guts," Acker says. "That was just an inconceivable no-no
to me. That pretty much trashed our friendship. Forever and ever." Forty
years later you can still hear the intensity in Acker's voice. "When you
get in trouble, you don't squeal on anybody."
Charlie Schulz and Ken Hopper, members of the technical staff of the Telephone
Crime Lab at Bell Laboratories.
Hopper's path to the Telephone Crime Lab was a circuitous one. In 1971 he
was a distinguished-looking forty-five-year-old electrical engineer, a bit
on the heavy side, with blue eyes, short brown hair, and glasses. Hopper
had joined the Bell System some twenty-five years earlier, shortly after
the end of World War II. Within a few years he had found himself at Bell
Laboratories' Special Systems Group working on government electronics projects.
The stereotype of government work is that it's boring, but Hopper was a lightning
rod for geek adventure: wherever he went to do technical things physical
danger never seemed far behind. There was the time he had to shoot a polar
bear that had broken into his cabin while he was stationed up in the Arctic
working on the then secret Distant Early Warning Line, the 1950s-era radar
system that would provide advance warning of a Soviet bomber attack. Or the
time he almost died in a cornfield in Iowa while building a giant radio antenna
for a 55-kilowatt transmitter to "heat up the ionosphere" for another secret
project. Then there's the stuff he still can't really talk about in detail,
involving submarines and special tape recorders and undersea wiretaps of
Soviet communications cables.
The Special Systems Group was a natural to help AT&T with the Greenstar
toll-fraud surveillance network in the 1960s, Hopper says, and that work
led to involvement with other telephone security matters. But the Telephone
Crime Lab also owes its existence to the FBI. Hopper recalls, "In the mid-1960s
the FBI laboratory came to our upper management and said they were getting
electronic-involved crimes. They had no people in their laboratory that could
examine evidence in these cases, especially related to communication systems,
and they asked for Bell Labs' assistance. Upper management of Bell Labs agreed
that this was in the public interest and that we would do that. The work
was assigned to my organization, Charlie Schulz being the supervisor. We
had just a few people, never more than two or three, working on this stuff.
The Ashley-Gravitt affair was much in the newspapers that fall and attracted
the attention of Louis Rose, an investigative reporter at the St. Louis
Post-Dispatch, Missouri's preeminent newspaper. Rose had written a series
of articles examining the apparently cozy relationship between Southwestern
Bell and the Missouri Public Service Commission, its regulator in that state.
"I had been looking at all the expenditures and all of the salaries and donations
by Southwestern Bell," Rose recalls. James Ashley, he says, "found a convenient
thing in me, because I was already looking up these ties."
In January 1975 the Texas scandal spread to North Carolina when a former
Southern Bell vice president -- another who had been forced out of the telephone
company, as it happened -- admitted during an interview that he had run a
$12,000-a-year political kickback fund for the Bell System. The telephone
company soon found itself being investigated by an assortment of agencies:
the Securities and Exchange Commission, the Department of Justice, the Federal
Wiretap Commission, the FCC, and the Texas attorney general.
The next shoe to drop in the scandal was, in a way, predictable, so predictable,
in fact, that Bill Caming, AT&T's patrician attorney for privacy and
fraud matters, had predicted it ten years earlier. Caming couldn't say exactly
when it would happen, or exactly how it would happen, but he was sure it
would happen. Ever since I965, when he had first learned about AT&T's
Greenstar toll-fraud surveillance system, with its tape recordings of millions
of long-distance calls and its racks of monitoring equipment kept behind
locked cages in telephone company central offices, Caming had maintained
it was a matter of when -- and not if -- the news of Greenstar would eventually
The "when" turned out to be February 2, 1975. The "how" was a front-page
headline in the St. Louis Post-Dispatch: "Bell Secretly Monitored
Millions of Toll Calls." The article, by Louis Rose, quoted an anonymous
source within the phone company and was chock-full of details: a list of
the cities where Greenstar had been installed, the specifics of its operation,
the stunning news that the phone company had monitored 30 million calls and
tape-recorded some 1.5 million of them. Someone -- someone high up, it seemed
-- had spilled the beans. By the next day the story had been picked up by
the newswires and the New York Times.
Caming didn't need a crystal ball to predict what happened next: a phone
call from the chair of the House Subcommittee on Courts, Civil Liberties,
and the Administration of Justice. "He said. 'I think we're going to have
to have one of your guys come down and explain all this to us," Caming knew,
as he had known for ten years now, that he would be the guy.
Less than three weeks later Caming found himself before the U.S. Congress.
swearing to tell the truth, the whole truth, and nothing but the truth. Seated
with Caming were Earl Conners, chief of security for Chesapeake and Potomac
Telephone Company, and John Mack, a Bell Labs engineer who was intimately
familiar with the technical details of Greenstar. True to his reputation
for loquaciousness (or maybe it was his legal training) Caming made sure
his colleagues never got to speak more than two dozen words over the course
of the three-hour hearing. Caming explained AT&T's motivations for launching
the surveillance system, how it operated, and, most important, why it was
legal -- indeed, not just legal, but in fact the only option AT&T had
to combat blue box and black box fraud at the time. Never once did he refer
to it as "Greenstar," the name that ten years earlier he said "just sounds
illegal." Perhaps it was Caming's legal reasoning, perhaps it was his appearance
-- competent, prepared, confident, yet self-effacing -- or perhaps it was
195 Broadway's deft handling of the press on the matter, but AT&T managed
to weather the Greenstar storm without much damage. Despite some alarming
headlines there was little fallout and no criminal investigation. The Greenstar
matter quickly faded away.
95 "decline to prosecute": Rose, "Bell Secretly Monitored Millions
of Toll Calls."
96 "Change the name": During my interviews with Bill Caming I often
used the term Greenstar in our discussions. Ever the AT&T attorney, he
would periodically correct me: "No, that's not its name. That was an internal
code name that we stopped using." Sometime later I visited the AT&T Archives
in Warren, New Jersey, which maintains a computerized index of old Bell System
files. I typed in "Greenstar" and watched the display light up like a Christmas
tree as it found relevant documents. When I mentioned this to Caming a few
days later, he gave a rueful laugh and responded, "Well, I guess you can't
keep a good name down."
96 two parts to Caming's reasoning: Before 1968, the federal wiretapping
law was Section 605 of Title 18 of the United States Code. It was a strangely
written law. As discussed in the next chapter, section 605 did not make
wiretapping ("interception") itself illegal. Rather, to commit a crime under
605 you had to both intercept a communication and then disclose the contents
of the communication to someone else. Clearly when Greenstar recorded a call
and a human listened to it, there was an interception, but because the trained
operator listening to the tapes never discussed the contents of the communication
(just the signaling of the call itself), there was no disclosure, and thus,
AT&T asserted, no crime. In 1968 the Omnibus Crime Control and Safe Streets
Act became the new law that governed wiretapping -- but that law had specific
carve outs for random monitoring and interception of communications by telephone
company personnel attempting to protect the assets of the telephone company.
96 "imprimatur": Caming, "Surveillance," pp. 243-44.
96 Congressional Research Service: Ibid., p. 234.
97 "Lonely Boy": "Lonely Boy Devises Way of Placing Free Long Distance