1. They wanted to do a new website and despite being advised against it, decided to co-locate it cheaply on a popular hosted WordPress platform.
2. They did not have direct contact with the co-location company, as the account was set up through the developer they hired to build the site.
3. The site was done poorly and required some strange DNS entries to make it work.
The story goes as follows:
We get a call that email messages are getting bounced back. I look at the bounce-back message and see that they have been blacklisted. I go into my IR plan for blacklists and check the appropriate channels to see how bad it is. My first check at MXToolbox.com shows 21 listings, but for an IP address not assigned to my customer's netblock. Curious, I look at who owns that block and find that it's the hosting company I specifically advised against about 6 months prior. I then check the customer's mail server IP and see that it also blacklisted, and that every IP listed in the DNS record has been listed.
I will list out my IR plan for this occurrence in a later post as it is now again a work in progress after I discovered some flaws while going through this incident. My basic steps were:
1: Check firewall for large amounts of SMTP traffic. (This was clear)
2: Check exchange queues for large amounts of mail waiting to be delivered, large amounts of connections. (Also Clear)
3: Check firewall rules set up for port 25 and 987. (Again Clear)
- From these steps I determined that the problem was external-
4: Check MxToolbox blacklist (Bingo)
5: Check known SaaS spam blocking solutions - TrendMicro, McAfee, Reflexion, Etc. (All had both IP's listed)
6: Once I determined that the SPAM was coming from the web server, and thanks to the brilliance of MXToolbox it was even possible to determine the site that had been compromised on the co located server, we called the customer to advise him to reach out to the hosting provider to request that the other site be shut down until it could be cleaned. Here is where it got ugly:
The could not call directly to the provider as they did not have an account. The web developer (I use this term very loosely, as slinging WordPress isn't technically developing) denied that it was their problem, and was basing his opinion on a search of the wrong domain name. He blamed a DNS problem because he couldn't be bothered to check the email address they use to get the correct info. Between his denial and slowness to respond, it took nearly half a day to get him to call the hosting company and begin to get the issue cleaned up. One he finally relented and the SPAM stopped, I began the long process of de listing the IP's in all of the blacklists that had self-service options. It then took a full two weeks to get the all clear from TrendMicro, as they have some strange procedures around the de listing process.
So what did we learn?
1. Cheap hosting provides minimal security (Spoiler Alert - Cloud options are the same...)
2. WordPress is as easy to compromise as it's been described to be. Co located servers mean you never know when a popped plugin is running on the same environment as you are.
3. Always CYA (Check Your Assumptions)
4. Every customer needs a detailed and customized IR plan tailored to their environment. No two are the same, and every process is different. If you're not "singing off the same sheet music" it's hard to stay together as you move through the steps.
What was the impact to the customer?
1. Loss of ability to communicate with his clients for up to 2 weeks
2. Couldn't email invoices. They spent hours per day faxing invoices so that they could continue to get paid.
3. Lost time and employee dissent.
All in all, it didn't matter what firewall rules I had in place, how solid my security was on their workstations, or even how well done their site was, because they chose a bad hosting platform and another site got popped. It also demonstrated to them how critical email has become to a business and motivated them to take some additional steps to help mitigate these problems in the future.
One last thing: They saved around $20 per month over a dedicated virtual server with an unshared IP.
