26 April 2015
Published here to resist censorship.
Surveillance system used for censorship in Europe
Censorship attack combines packet injection and Heartbleed
We all know there is censorship online. It happens in China. It happens to
"terrorists". But we don't believe it will happen to us.
As Eben Moglen and Kaspersky have pointed out, companies developing
crypto are prime targets no matter where they are. So you don't have to be
a bad guy for the NSA to attack you. You just have to protect people from
the NSA. Even protecting yourself is often enough. NSA prefers their victims
to be defenseless.
Detection in the wild
In early 2015 people were still downloading our ISO file for GoodCrypto.
But suddenly installations stopped.
After a lot of checking we noticed that the downloads got HTTP 200 result
codes, but the lengths were all too short. This isn't supposed to happen.
A 200 result means success. These weren't successful downloads, but the web
logs said they were. Ordinary log checks didn't show the bug.
Finding the vuln
Downloads from goodcrypto.com to goodcrypto.com worked. Downloads from another
site at a different datacenter in the same country worked. A little further
away in the network, downloads failed but the server logged a "Success" status
The obvious answer was a server misconfiguration. We couldn't find one. A
server side packet dump showed the client just dropped the connection in
the middle of the download.
We couldn't get a browser to download the whole ISO file. The browser thought
it came in fine, but the file was incomplete.
So was it a browser bug? We tried other browsers. They couldn't download
The wget program often helps debug downloads. It doesn't have the same malware
issues as browsers, because wget doesn't support malware vectors such as
you why it failed.
When we tried wget, it detected errors, retried, and finally succeeded. It
said the error was a bad length field in a TLS packet. That didn't make sense
at first because we thought TLS packets were error corrected by TCP.
We searched for other bug reports like this. They were all during session
initiation, not in the middle of a long download.
But our searching led to Heartbleed. Modifying SSL/TLS length fields is exactly
how it works.
Wasn't Heartbleed fixed in 2014?
In 2014 when we all heard about Heartbleed many servers were vulnerable.
But OS providers fixed it fast. Our own servers get regular security updates.
How were we seeing it now?
Servers around the world were fixed fast, but clients were vulnerable too.
The Heartbleed news coverage was all about servers. Servers got fixed. Many
clients didn't. A client side Heartbleed attack is sometimes called Reverse
Packet evidence shows MITM
Was our server cracked? We're pretty careful, so that doesn't happen often.
But we checked, and checked again. Even though we don't usually have any
packet logs, we ran download tests with simultaneous packet logs on both
the server and client.
The server packet logs showed an ordinary number of bad packets, all error
corrected. During the download the client dropped the connection.
The client packet logs were very different. There was a surprising number
of bad incoming packets. Almost none of these bad packets showed in the server
logs for the same session. These packets appeared to be injected into the
packet stream. This is an MITM/MOTS attack, specifically a packet injection
Finally wget reported that a TLS packet with a bad length field got through
and caused the TLS connection to break. Now we knew this was a MITM variant
of Reverse Heartbleed.
Working around the censorship
We added simple instructions to our Download page:
Somebody (Hi NSA!) is trying to censor GoodCrypto downloads.
But don't worry. The workaround for their super duper advanced network attack
is: Just use wget.
If you're on windows you need wget.exe.
At a command prompt:
wget --no-check-certificate (URL appeared here)
Because you will verify the file hash, "--no-check-certificate" is ok in
this rare case, and Windows needs it.
We also strengthened the encouragement to verify the file, and asked visitors
to let us know when the attackers change tactics.
The attacker injected packets by forging our site's IP address. If a site
visitor notices the packets, they will think that the attack is from us.
The attacker didn't just shift blame away from themselves. They framed someone
they don't like. Faking the evidence to blame an attack on someone else,
especially someone the attacker doesn't like, is a classic False Flag operation.
An MITM attack like this requires impersonation, so the benefits to the attacker
of a False Flag are built right in.
One way to censor anyone is to attack their reputation.
Censorship by a nation state
Even though the download broke, the browser didn't complain. It looked like
a successful download. The server showed an HTTP 200 result code. Neither
the client nor server detected the attack.
Server packet logs didn't show anything unusual except the abrupt client
disconnection. The attack didn't show at all during downloads from a client
that was close to the server in network space, such as another nearby datacenter.
It wasn't an attack on the server or the hosting provider.
We set up another server in a different country and the attack continued.
It wasn't an attack by the hosting country. We were nowhere near our transfer
limits for either server, so it was not traffic shaping by hosting providers.
The clients were all over the world. It wasn't separate attacks launched
from close to individual clients.
The attack appears to be from someone sitting on the net pipes and injecting
packets. This requires huge resources. The U.S., U.K., Canadian, and Chinese
spooks do this. Ordinary criminals don't have the ability.
Because the goodcrypto.com servers are in Europe, China is an unlikely suspect.
GCHQ and CSE are dependent on the U.S. for their QUANTUM capability. That
means it was likely an NSA attack, either directly or by proxy.
But which nation really doesn't matter. There is no known way to protect
against specific nations. You have to protect against all or none.
Who is vulnerable
Anyone who publishes on the web is vulnerable to this form of censorship.
Even if you just use HTTP, the Chinese censorship method of a simple RST
Update: The attack is now intermittent. Exposing them often helps. When it's
paused they may be avoiding forensics, or just changing tactics. Let us know.