My 2014 Year in Review

So after a really crazy year, I felt like it would be good for me to look back and recap what I took out of 2014, if only for myself.  Here's my themes and lessons:

1. Open Source is not an excuse for bad security:

I took on a few Linux projects in 2014 and saw quite a few servers not behind an IPS because they were running IPtables.  No one took the threat seriously because open source software was trusted and seen as secure.  Heartbleed and Shellshock changed that, and I for one am glad it did.  There's no longer an argument that open source can be trusted implicitly, and i talked several customers into putting a WAF in front of these boxes and setting up rules and IPS properly because of these large scale vulnerabilities.

2. Patching is not the only game in town:

Waiting for a patch, and then applying that patch is slow and tedious.  Mitigating risks quickly is what my employer is paid to do, and we have to go above and beyond with different approaches to emerging threats.  This leads me to:

3. Think outside the box, and use the tools you have:

When CryptoWall was spreading through ad networks, we used content filtering to block those ad networks instead of waiting for our AV vendor to put out a definition. Content filtering is not made for this role, but it did work well and we avoided CryptoWall almost entirely.

4. Not all threats require action:

We found that Poodle wasn't a big deal for our environments.  Once I understood the threat vector and how it was being exploited, I found that the fix for the issue was not worth the huge disruption in service that the implementation would require.  There were very few systems actually negotiating SSLv3 sessions, and those that were could usually be shut down.  The short version is, I no longer accept a threat with a high CVSS score as requiring a patch until I run it through my threat modeling regimen.  It's not easy, it's not simple, and it's not perfect.  What it has become is a way of communicating quickly what systems could be affected and the business case/ impact around the vulnerability.

5. Knowing your systems is the best way to protect your systems:

Know what you have and you'll easily figure out how to protect it.  We use a monitoring and management tool built for MSP's that quickly lets me build reports based on installed software or patch status or whatever you might want to know.  This is a key to my threat modeling system I am building, and I will tell you that I have found quite a few packages that didn't need to be there and could be removed, including a Joomla site that had been compromised and only served as a link to a file share. The rest of it gets built into my database of implemented technologies so I can search quickly when a certain package or protocol is found to be vulnerable.  Inventories are really great, but great inventories are really important.

What was your biggest takeaway from 2014?

My recent bout with a WordPress Vulnerability

We all hear horror stories about WordPress sites, and for one of my customers, it finally came to pass two weeks ago.  It isn't until today, two weeks later that we have full functionality restored to their systems. Let me set the stage:

    1. They wanted to do a new website and despite being advised against it, decided to co-locate it cheaply on a popular hosted WordPress platform.
    2. They did not have direct contact with the co-location company, as the account was set up through the developer they hired to build the site.
    3. The site was done poorly and required some strange DNS entries to make it work.

The story goes as follows:
We get a call that email messages are getting bounced back.  I look at the bounce-back message and see that they have been blacklisted.  I go into my IR plan for blacklists and check the appropriate channels to see how bad it is.  My first check at MXToolbox.com shows 21 listings, but for an IP address not assigned to my customer's netblock.  Curious, I look at who owns that block and find that it's the hosting company I specifically advised against about 6 months prior.  I then check the customer's mail server IP and see that it also blacklisted, and that every IP listed in the DNS record has been listed. 

I will list out my IR plan for this occurrence in a later post as it is now again a work in progress after I discovered some flaws while going through this incident.  My basic steps were:

1: Check firewall for large amounts of SMTP traffic. (This was clear)
2: Check exchange queues for large amounts of mail waiting to be delivered, large amounts of connections. (Also Clear)
3: Check firewall rules set up for port 25 and 987.  (Again Clear)
      - From these steps I determined that the problem was external-   

4: Check MxToolbox blacklist  (Bingo)
5: Check known SaaS spam blocking solutions - TrendMicro, McAfee, Reflexion, Etc.  (All had both IP's listed)

6: Once I determined that the SPAM was coming from the web server, and thanks to the brilliance of MXToolbox it was even possible to determine the site that had been compromised on the co located server, we called the customer to advise him to reach out to the hosting provider to request that the other site be shut down until it could be cleaned.  Here is where it got ugly:

The could not call directly to the provider as they did not have an account.  The web developer (I use this term very loosely, as slinging WordPress isn't technically developing) denied that it was their problem, and was basing his opinion on a search of the wrong domain name.  He blamed a DNS problem because he couldn't be bothered to check the email address they use to get the correct info.  Between his denial and slowness to respond, it took nearly half a day to get him to call the hosting company and begin to get the issue cleaned up.  One he finally relented and the SPAM stopped, I began the long process of de listing the IP's in all of the blacklists that had self-service options.  It then took a full two weeks to get the all clear from TrendMicro, as they have some strange procedures around the de listing process.

So what did we learn?

1. Cheap hosting provides minimal security (Spoiler Alert - Cloud options are the same...)
2. WordPress is as easy to compromise as it's been described to be.  Co located servers mean you never know when a popped plugin is running on the same environment as you are.
3. Always CYA (Check Your Assumptions)
4. Every customer needs a detailed and customized IR plan tailored to their environment.  No two are the same, and every process is different.  If you're not "singing off the same sheet music" it's hard to stay together as you move through the steps.

What was the impact to the customer?

1. Loss of ability to communicate with his clients for up to 2 weeks
2. Couldn't email invoices.  They spent hours per day faxing invoices so that they could continue to get paid.
3. Lost time and employee dissent.

All in all, it didn't matter what firewall rules I had in place, how solid my security was on their workstations, or even how well done their site was, because they chose a bad hosting platform and another site got popped.  It also demonstrated to them how critical email has become to a business and motivated them to take some additional steps to help mitigate these problems in the future.

One last thing:  They saved around $20 per month over a dedicated virtual server with an unshared IP.

Diamonds from Dust- Secure the Edge

First Things First - Content Filtering
Firewall configuration should have been a single post, but I realized that I do quite a lot of configuration without even thinking about it.  With that said, I decided to break this up into small but progressively more complicated tasks.

One of the things I see disabled almost immediately by most firewall admins is content filtering.  Yes it is overbearing, annoying, and raises concerns over censorship.  What it can do is protect your workstations from crap floating around the internet embedded in sites they really had no reason to be on in the first place.  Here are some tips for keeping your job, and keeping people on your side:

1. Go slowly:  If you don't have content filtering turned on, configure the policy to only block sites in categories such as freeware or malware.  Then enable content filtering and see if anything breaks.

Next, print off the list of categories that are available in your platform, and take it to the leadership of your organization to get approval on which categories you can block and which are off limits.  Establish a process for white listing sites that are miscategorized or are necessary for business. I like to set up a system where I can email the decision maker and only white list a site with their approval in an email.

Once you have your marching orders, enable one category per day and wait for the calls to come in.

2. Explain the change in advance, and let leadership take the heat for the decision.

3. Work with users, don't make them feel like you're working against them.

Advanced Categories and Custom Configs:

I will shout it out on top of a mountain: "I HATE ADS"

I also have found that it's very easy to block them by loading the frame in a seperate window and grabbing the URL.  I then add it to my blacklist and say goodbye to the Russian Brides and the free games.  If the Zedo issue from earlier this week taught us anything, it's that no good comes from ads.

The other category that I really like but that breaks a lot of sites is the "Uncategorized" category.  Be advised, your life will be hell for a while.  But after some tweaking and some grumbling, our Malware infections at customer sites that have allowed me to implement this had plummeted. One customer had 5 infections in a week, and since we put the work into this category it's dropped to one in 9 months.

Finally, stick to your guns.  This will likely be a painful change for a lot of users but it will be worth it in the long run.

SMB Infosec -Diamonds from Dust Part 1

This is the first blog in what I expect will become a series of blogs on methods I have come up with for securing the SMB company with limited resources.

Diamonds from Dust will be written primarily from an MSP's point of view. I firmly believe that every small business needs a partner with some technical resources in order to remain secure. This isn't about an onsite person's technical abilities at all, it's about economies of scale.

This post is about selecting a vendor. This will be the only post written from a customer's perspective.

Here we go!

The 5 things your MSP must have:

1. Technical Certification:  Any MSP that has been around a while will have certified engineers on hand. This is to meet requirements for vendor recognition programs. Vendor recognition drives prices down and service up. You want that ability in your MSP.

2. A Monitoring Solution (with custom rules): Monitoring is critical, and a robust monitoring solution is critical for insight. A well tuned monitoring solution is key to your success and security.

3. A Security Focus:  Your MSP must be interested and immersed in security. It must care and it must respond to threats.

4. A Customer Notification System:  You need to be kept in the loop when things are bad and a way to rapidly notify customers is critical.

5. A Solid BCDR Solution: This will be the subject of at least 2 more posts in the coming weeks and months.

Next Post: Minimum Firewall Settings

Other Topics Coming:
BCDR
Monitoring Rules
Managing AV
Incident Response
Planning

InfoSec Leadership - Creating a culture and avoiding gimmicks

Intro:
An organization's identity and culture is the source of pride for many, and a source of many problems for others.  I have been studying Organizational Leadership for quite a while now, and have come to realize that IT in general, and Infosec in particular is mostly getting it wrong.

Why?
In general, IT has been promoting the best technical people into roles of leadership.  Apart from the C-level positions, your average IT organization ignores leadership skills and rewards those who perform technically well.  This generates a great deal of referent power, but ignores the personality characteristics that must come with a position of leadership.

With a higher position comes reward power, coercive power, and several other sources of power that a leader has at their disposal.  Gaining these tools in leadership can be an issue for those not prepared to lead.

In working with many IT people over many years, I see a lot of "Power Drunk" senior staff wielding coercive power around and mentally beating down junior staff for simple, and honest mistakes.

This is not how we lead.

This is how to alienate people.

This breeds more destructive behavior.


On the other end, when dealing with users many companies implement reward programs, which reinforce good behavior.  Unfortunately, it only works on people who want the rewards, and  only until they have gained the rewards.  Reward, or bounty systems have a severely limited shelf life and are challenged by limited efficiency.


How does this apply outside of IT?

Infosec is typically the most important policy surrounding the IT department for a company to enforce.  As we saw above, we have oversimplified leadership into two types of programs:

Coercive - Don't screw up or you're fired.  Even 2 strike policies don't work.

Reward - The gamification of Infosec suffers the problems i outlined above.

Infosec lacks true leadership:

As tech people we should know that there's no simple way to solve a complex problem.  So why do we oversimplify leadership across internal organizations?  It's simple, they don't like us.  We lack the personal power, which would serve as a source of balance in our leadership practices.

Avoiding Leadership Gaps:

1. Hire personable people within your Infosec group - Also, make sure they are visible within the organization and are likable when dealing with users.  People are more likely to listen to people they like.

2. Be transparent - Give the organization a sense of what you're doing to help protect them and to make their jobs easier.  Make sure that you reinforce that it is an organizational effort, and get users to buy in through a sense of working together.

3. Build a comprehensive game plan - If you want to be a true leader in your organization, go to leadership with a way to build rapport.  You do not need to spend money, you just need to show you can blend sources of power into a change in culture, not just a policy.

Finally:

Work with what you have at your disposal.  Some people will be above policy, but no one is above culture.  Some people will get away with things they shouldn't, and there is nothing you can do to change that.  If all you have is policy, you have a finite tool that will never be applied evenly.  Culture has a strange way of weeding out those who don't fit in.  They lose support from those who see them as an outlier, and eventually fall out of favor with leadership.

More to come about leading an Infosec team next time.