Spam zombies and port scans - to log or not to log
By PotstickerGuru on Jan 27, 2005
It was exactly a year ago that the big ISP in B.C. offered broadband DSL along Highway 99 up to Whistler, and I was one of the first on that leg to subscribe. In just that year, I've seen some tightening up of packet filtering on the ISP's network. Within about a month after DSL became available, apparently, I picked up intrusion attempts by at least three other compromised systems from neighbours. I sent out some email to the home owners up there to be on the lookout for unusually heavy network activity on their routers when they weren't actively using their systems. And I wasn't the only one that noticed. About 6 months into our new found bandwidth, the ISP decided to shutdown free flow of port 25 SMTP traffic from any subscriber except through their mail routers.
Such action by my ISP was annoying, but easily circumvented by tunnelling packets through a virtual private channel back to my mail server back in California, so my problems were solved, and I still retained some autonomy and privacy. But, I'm sure that for quite a few customers, including some of my neighbours up there, the ISPs actions caused some grief. You see, for some of my neighbours, a significant fraction of their emails began to bounce and were no longer getting to their intended and legitimate recipients. Mail was bouncing due to the ISP's mail servers getting onto DNS blacklists as primary sources of SPAM. And the reason why the ISP's servers got blacklisted was that the spammers adapted to the block on packets destined to port 25 on non-ISP servers; they decided simply to route email through the ISP's mail gateways. And to avoid the ISP from tracking down all the compromised systems, the spammers didn't ust use a few spam-running-zombies, these folks compromised hundreds of systems and had each one send just a few thousand emails and then stop after a couple of days, until the next campaign. This caused half of the ISP's mail servers to get onto some of the major DNS blacklist servers out there, and I would guess their tech support guys had to field a lot of calls from folks that ended up with rejected emails and needed to switch SMTP gateways. Three or four out of the half dozen ISP's mail servers in the lower mainland B.C. and Alberta are currently or were as of a few weeks ago, on the top world's DNS blacklists for sending too much spam. In fact, I think more than 90% of email coming out of their network is spam, at least I block about 80 to 90 spams a day from them with no abatement and 99+% of that is spam. And while I've configured my mail server at home to block spams and return a polite spam Error 550 messag, the ISPs around the world that route spam emails often just seem to ignore, or worse, forward the problem, as opposed to aggressively dealing with the situation and solving it.
A clear example was a case of rejected spam which I tracked was originating from a poor guy in southern California, who evidently, suffered a fatal disk crash after I contacted him and told him about the problem. He ended up having to re-format and install his operating environment. How I found out his system was compromised and a spam zombie was quite a coincidence. A month and a half ago, during a 30 minute period, I received over 1000 emails from about 5 MTAs worldwide that were bouncing an undeliverable spam to me, the apparent sender. Fortunately for me, 4 of those 5 mail servers included the message with full headers, and clearly, I could tell that the first hops and last hops were not from my IP address domain. But the ISPs should have easily figured that out and just killed or dropped the email or simply denied mail routing because the mail headers and addresses were so obviously mismatched. But as I said, many ISPs are just sloppy about mail filtering and don't bother. So despite me obviously not being the true sender of the spam, these ISPs just let me have the flood of bounced emails. In fact, one ISP's automated SPAM fighting machine apparently recognized the 250+ emails it got as spam, but then decided to reply to the faked Sender address with some legal-mumbo-jumbo about abuse of terms of service. Geez. I felt like half these ISPs were just playing dumb and arrogant. Clearly, their own header information encapsulated within the email indicated it was a spam and the sender domain and MTA IP address had huge mismatches.
To stop the flood for the next few hours, I decided to simply block all emails to that address and send immediate errors messages that explained that this address was not valid. The campaign did only lasted about 2 hours, and then the numbers of messages subsided after several thousand bounces. How many were actually delivered, I don't know. It did make me think.
But the lucky coincidence for me was that all the spoofed emails were using an email alias that I publish to the network for just my fishing msgboards. And by inspecting the headers in some of the bounced emails, I quickly found a common point of origin from an IP address in southern California. And the two pieces of information led me to check my web server access logs, and I did get a match, plus a bonus piece of info: a cookie ID. This cookie is something I plant in my web pages that can help identify unique sessions, especially identifying HTTP connections for logins.
This allowed me to identify the actual user and again, luckily, he had contacted me in the past and left an email address. Unfortunately, he was a skilful angler, but not a big IT technician, and so he wasn't sure beyond running standard anti-virus software how he could stop being this spam zombie. And unfortunately for him, I guess the folks using his system for a zombie were finished and didn't want many traces of their activity. Within just a day after I notified him, his computer disk crashed and all data was lost. After a week of silence, he emailed me back and told me about the crash, and the subsequent re-format and re-install of his entire system.
All this bad network activity in the past month or so spurred me to turn on aggressive logging on most of my home server and router systems. So just two weeks ago, I started to get a rash of panic emails from my router up in B.C. Evidently, pings of death were being detected and I had set the system up to email me immediately. Again, like the spam incident I was getting copious emails, this time not as quickly, but they were averaging many per minute and they indicated that the attacks were coming from 7 separate networks in at least 3 different countries. For two days, the router logs were arriving in my mailbox here in California almost once every 30 seconds. I wrote the ISPs to politely forward abuse emails to the right folks in their network to stop the attacks on my hosts. Most did have an automatic mail responder, but only the Germans sent back a personal response to my inquiries and told me that they have identified the host and have forwarded the headers and logs to those in charge of that subnet for investigation. After 3 days, and a few megabytes of logs and emails, the pings of death and port scans finally stopped up in Canada. I'm not sure if rebooting the router and getting a new IP address assigned was the trick or if the campaign just stopped.
Being the curious kinda person I am, I couldn't just be satisfied with the status quo, so I decided to turn on aggressive packet filtering and logging on my local systems here in California. I have two servers that run 24/7 and have open interfaces to the internet. I do have firewalls turned on, but I was not logging the packet rejections or denials. So for the last few days, I decided to turn them on and observe. Just between 2:45am last night and 8am this morning, I rejected about 120 attempts on my mail/web server, and about 40 attempts on my NAT firewall box. I have funky ICMP packet requests that don't look like pings of death. I have strange UDP and TCP attempts at really weird high port numbers that don't conform to any service or standard, and by far, the 90+% of denied packets are port scans for 139 and 445 NTFS file share UDP ports. And that was just in a little over 5 hours. It's incredible to me just how many scripts set off by hackers there are out there, and how many unique attempts occur to gain unauthorized remote entry onto a system there are. The costs must be staggering for folks with systems less robust and less protected against these hackers. But just having that knowledge itself can be pretty depressing, especially seeing how it means we need to be ever more vigilant against intrusion. It's almost enough to turn off the logging, save some disk space, and just live in ignorant bliss for a (short) while.