Today we noticed we that a lot of our incoming email was being marked as spam. Since this included many messages that were clearly not spam, we decided to investigate more deeply. We found that a third party was subverting the Spamhaus service using a routing-based attack.
First clues: Spamhaus misbehaving?
To filter out spam we use a combination of about 30 different rules. One of them is based on the Spamhaus advisory services, which are a family of DNS-based block lists (DNSBLs) commonly used to filter email. Spamhaus keeps track of servers which are sending large amounts of spam, and publishes the IP addresses of these servers. In our system, when an email comes in, we use Spamhaus to look up the IP address of the remote server. If we find a listing, we compute that the chances are higher that the message is spam.
As each incoming email goes through our system, we record which anti-spam rules contributed to its overall spam score. Today there were many more points attributed to Spamhaus advice then usual, so we suspected something was wrong with Spamhaus or that they changed their systems in some major way. But we checked with them, and found nothing had changed. Even more surprising, the IP addresses our system was recognizing as "listed in Spamhaus" did not have official records on the Spamhaus web site.
Spamhaus is checked using DNS, so we can use simple DNS client tools to make queries by hand. For example, if an email is received from IP address 24.43.30.128, we check whether that address exists in the Spamhaus "ZEN" list on the UNIX command line as follows:
1
2
$ host 128.30.43.24.zen.spamhaus.org
128.30.43.24.zen.spamhaus.org has address 127.0.0.2
A reply (always in the form 127.x.x.x) means that there is a positive match in the database.
Spamhaus presents 22 public DNS servers for querying their system. After some testing, we discovered that one of these (ns0.spamhaus.org, 204.16.254.40) was always returning "127.0.0.2" to any query. So it seemed one of their systems was serving bad data, reporting that every IP address on the internet was a source of spam. This meant about 5% (1/22) of our incoming emails were getting a much higher spam score than they deserved, and thus a larger proportion of messages were being marked as spam.
Locating a rogue Spamhaus server
As Spamhaus has a good track record, we thought this behaviour was uncharacteristic. An error like this would normally be fixed quickly. So we suspected something more complex was going on. We looked up the server in the RIPE database and found it was in a group of addresses controlled by Vitalwerks Internet Solutions. Then we performed a traceroute to the address, which produced some strikingly unusual results:
1
2
3
4
5
6
7
8
# traceroute -n 0.ns.spamhaus.org
traceroute to 0.ns.spamhaus.org (204.16.254.40), 30 hops max, 60 byte packets
1 xxx.xxx.xxx.xxx 0.213 ms 0.189 ms 0.150 ms
2 xxx.xxx.xxx.xxx 0.945 ms 0.971 ms 0.945 ms
3 * * *
4 * * *
5 * * *
6 204.16.254.40 17.831 ms 17.752 ms 17.716 ms
Several 'hops' along the path (3, 4 and 5) did not reveal themselves to the traceroute at all. We expected to see some hops belonging to Vitalwerks. One 'missing' hop is not very unusual, but several consecutive hops (almost the entire route!) being missing was another clue to something fishy. We had to look more deeply, by asking our BGP routers, to find out where this misbehaving Spamhaus server was really located:
1
2
3
4
5
6
7
8
9
router# show ip bgp 204.16.254.40
BGP routing table entry for 204.16.254.40__/32__
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Advertised to non peer-group peers:
xxx.xxx.xxx.xxx
__34109__ 51787 1198
193.239.116.204 from 193.239.116.204 (84.22.127.134)
Origin IGP, metric 10, localpref 140, valid, external, best
Last update: Tue Jan 5 11:57:27 1971
Finding the truth, getting back to normal
As can seen from the BGP output, we were using a /32 route going over AS 34109. This was highly suspicious for two reasons. First, a /32 route refers only to a single IP address. Except in special cases, routes are normally /24 (256 hosts) or larger. Second, the AS 34109 belongs to CB3ROB which is an Internet provider that has actually been in conflict with Spamhaus (see: spamhaus; allspammedup; theregister). Certainly they weren't running a legitimate Spamhaus server.
It seems clear that the CB3ROB network hijacked one (or more) of the IP addresses of Spamhaus, and installed a DNS server there which incorrectly returns positive results to every query. The result causes harm to Spamhaus users and their customers, making Spamhaus unusable for anyone unable to correct the problem as we did, and perhaps even undermining the credibility of Spamhaus itself.
As we take measures to protect our routers from route hijacking, we were surprised to find this rogue entry appeared in our routing tables. We discovered a small misconfiguration in our software, and after fixing that the problem was solved. We could once again reach the real 0.ns.spamhaus.org server:
1
2
3
4
5
6
7
8
9
10
11
# traceroute -q1 0.ns.spamhaus.org
traceroute to 0.ns.spamhaus.org (204.16.254.40), 30 hops max, 60 byte packets
1 xxx.xxx.xxx.xxx 0.190 ms
2 xxx.xxx.xxx.xxx 0.394 ms
3 xxx.xxx.xxx.xxx 10.967 ms
4 r22.amstnl02.nl.bb.gin.ntt.net (195.69.144.36) 1.961 ms
5 ae-2.r03.amstnl02.nl.bb.gin.ntt.net (129.250.2.211) 3.695 ms
6 xe-3-0-3.ar1.ams3.nl.nlayer.net (69.22.139.202) 3.700 ms
7 as23352.vlan-102.ar1.ams3.nl.nlayer.net (69.22.139.123) 2.562 ms
8 ge0-4.aggrB3.ams3.nl.scnet.net (205.234.220.231) 3.953 ms
9 204.16.254.40 (204.16.254.40) 2.393 ms
Route hijacking has happened before, such as when Pakistan Telecom started announcing itself as the route to YouTube in 2008, but it is still rather unusual. As the core routing protocol of the Internet, playing "tricks" with BGP can be hazardous to the whole network. Moreover, Internet peering (route exchanging) relationships are based on building trust, so we don't think BGP is the place to make political demonstrations that cause network interruptions. We need to build trust between small network participants like Greenhost and CB3ROB, to foster digital diversity and innovation. But for now, we have suspended our BGP peering with the network that caused us all this trouble today — and our email services are back to normal.