Security mindsets of MSPs
-
About 18 months ago I took out a fairly standard 3rd line support and holiday cover contract with a local MSP to cover for my one-man-band operation when I'm not here. This is a pretty big MSP, employing dozens of people and turning over millions. As part of the contract, an engineer spent a day documenting our systems, and as part of that audit I gave him pretty much all the information someone would need to come in and manage our environment.
The results of this audit were written up in a Word document and included domain admin passwords, wi-fi/AP passwords, server names and configurations - pretty much everything. The company then e-mailed this document to me, in a completely unsecured e-mail. I e-mailed them back saying I couldn't believe they'd send the domain admin password in an unsecured e-mail and they just replied that they do this for all their clients and I am the first one to ever say anything.
I had to spend the day changing all of our environment passwords.
A few years ago, I employed another MSP. This was small, five man operation. Whilst I was away, one of their engineers was providing telephone to support to one of our users and gave the user the domain admin password in order for him to do something. When I returned and found out about it (the user told me), I asked the engineer if he thought that was good practice and was told that it was ok because the user wouldn't remember it.
I no longer trust external parties with my data. There are too many engineers out there who just don't give a shit about client security, either through laziness or ignorance. Which is a real ball-ache because it makes life harder when I am on holiday. I've created separate domain admin accounts for external support parties, and I keep these accounts disabled until they are needed. But since they are normally only needed when I am not here, enabling them is a problem. They also have their own logmein account running on one of our servers. I understand that cached credentials were vulnerable to Heartbleed, but I don't know if they cached credentials. I suspect they have and haven't bothered to tell me to change passwords (as per Logmein's recent advice).
I despair. If a Microsoft Gold partner, HP Gold specialist and VMWare Enterprise partner thinks it's normal practice to e-mail Domain Admin and wireless access passwords to clients in an unsecured e-mail, what hope do I have?
ARRRGGG
Now I know that most of you are great MSPs, so I'm interested in your best practices. How do you keep client details secure, and how to do you satisfy your clients that you know what you're doing. I still have the problem of how to give external support companies remote administrator access to our servers whilst I'm on holiday. At the moment, I have set-up a domain admin account for a colleague and trained her how to enable and then disable the admin accounts for our external support engineers via AD. I also have the domain admin password printed out and stored in the safe, which about five people here have the key to. I'm working on the premise now to "trust no-one", but still be able to take a holiday.
-
Before talking about how MSPs behave....
What you need for you concern internally is called a "break glass" system. What that is (I actually just learned this term although obviously not the idea) where an MSP can access a password that you provide (or access a system with a password) that cannot be done without setting off alarms.
For example, you have a system to which they have access that delivers a sealed password for emergency access. Upon doing so, it is logged, you are alerted, etc. And they have to answer as to why the glass was broken and when they are done, you reset the glass.
This allows you to provide access for emergencies without granting the access directly. I.e. They get access to get access, they don't just have access.
-
Most MSPs act this way for three reasons:
-
If you go with primarily "gold" partners, that means they are resellers not MSPs. Their specialty is sales, not IT. They are stores. To get to those levels you don't have to be technical, you have to sell gear. So you bar for what good looks like is actually almost impossible for a quality IT firm to hit but trivial for a non-technical salesman to hit. So you are possibly weeding out the secure firms and looking specifically to the insecure ones.
-
Most customers train MSPs to behave this way. It's true. Customers not only generally don't care, they often actively demand that MSPs be insecure. That someone wants security is often a massive shock to an MSP.
-
The MSP should only ever, when possible, have their own passwords. Not always possible, but generally. For example, their own domain admin account just like they were a member of staff. Because you need to know when something is done with your account or with theirs. Then they could only give out their own credentials for which their are liable instead of giving out hours for which they are not.
-
-
A break glass system (nice term, btw) wouldn't prevent unauthorised access as a result of stolen credentials though. By the time I get the alarm, it is probably too late to do anything.
-
@Carnival-Boy said:
A break glass system (nice term, btw) wouldn't prevent unauthorised access as a result of stolen credentials though. By the time I get the alarm, it is probably too late to do anything.
Nothing will prevent that. The idea is to prevent them being stolen.
-
I'm not sure I follow you. What's the purpose of the break-glass system? They still have full access, so it is just as insecure, right?
-
@Carnival-Boy said:
I'm not sure I follow you. What's the purpose of the break-glass system? They still have full access, so it is just as insecure, right?
One of the immutable laws of security is that you must always trust your admin(s). Period, no except. Immutable law. Anyone who can act as an admin has unlimited access. That can't be changed.
So given that, we must do what we can to limit this. There are a few basics:
- Audit everything that yo can (if necessary, be warned that auditing is only so useful and can be very counterproductive.)
- Be careful whom you trust.
- Limit the need to have an admin (break glass, in this example.)
In the case of the break glass, they only become an admin when necessary. Yes, they could break the glass when not needed, but you'd know about it. There is little alternative.
-
One of the most important documents that any IT pro can know....
-
Law #6: A computer is only as secure as the administrator is trustworthy.
-
I trust them to work on my systems. I don't trust them to prevent credentials from getting into the wrong hands as a result of their sloppy security procedures.
At the moment they can only come in if I let them, by enabling their AD account. The problem is who let's them in when I'm not here. If my colleague does this, how do I make it easy for my colleague? Alternatively, I could enable their accounts before I go on holiday, and disable them when I return.
In the days before remote access, they would have to physically enter the building to do any work. There are checks in place to prevent anyone walking in to the building (ie visitor management & security procedures). How do I replicate the physical management procedures in the on-line world? I need a robust system that balances the risk of unauthorised access versus the risk of downtime as a result of no-one being able to access something that needs fixing.
-
@Carnival-Boy said:
I trust them to work on my systems. I don't trust them to prevent credentials from getting into the wrong hands as a result of their sloppy security procedures.
You can't differentiate. Distill what you wrote to "I don't trust them."
So, you can't use them. That's your answer there.
-
@Carnival-Boy said:
At the moment they can only come in if I let them, by enabling their AD account. The problem is who let's them in when I'm not here. If my colleague does this, how do I make it easy for my colleague? Alternatively, I could enable their accounts before I go on holiday, and disable them when I return.
Yes. Or have management do it. Turn it on but have management hand over the passwords. Or have management enable/disable and nothing else.
-
@Carnival-Boy said:
In the days before remote access, they would have to physically enter the building to do any work. There are checks in place to prevent anyone walking in to the building (ie visitor management & security procedures). How do I replicate the physical management procedures in the on-line world? I need a robust system that balances the risk of unauthorised access versus the risk of downtime as a result of no-one being able to access something that needs fixing.
Well if the building was locked up and they needed to work.... either....
- Give them a key and they can get in any time or...
- Don't give them a key and trust someone internally with that right to let them in and out.
Technology doesn't change the choices. It might appear to, but it really doesn't.
-
@scottalanmiller said:
@Carnival-Boy said:
I trust them to work on my systems. I don't trust them to prevent credentials from getting into the wrong hands as a result of their sloppy security procedures.
You can't differentiate. Distill what you wrote to "I don't trust them."
So, you can't use them. That's your answer there.
I don't trust anyone external. But I have to take a holiday.
-
@scottalanmiller said:
Technology doesn't change the choices. It might appear to, but it really doesn't.
It doesn't change the choices but it changes the solutions. Creating a procedure that allows a colleague to grant admin privileges to external agents isn't a trivial task.
-
Them emailing you the passwords, for lack of a better phrase, is pretty derpy. Next go around, set up separate users and passwords for your main systems. That'll give you the ability to block their access as needed without having to hand over your passwords. If you want to get rid of them, toast their accounts. If you go rogue, they can still help your company keep control.
Also, take a look at the other types of companies that the MSP works with. Ones that have more security-focused clients will naturally lean towards being more secure.
-
Also realize that email may not be as insecure as you think.
Before you went off the handle and wasted all that time changing passwords, did you check if the email had been sent via TLS?
-
Went off the handle, LOL. Firstly, I don't consider changing passwords a waste of time. I probably don't change them enough. Secondly, it's the principle of the thing that annoys me. Bad practice is bad practice. I don't keep passwords listed in Word documents. I don't think they should either.
-
@JaredBusch said:
Also realize that email may not be as insecure as you think.
Before you went off the handle and wasted all that time changing passwords, did you check if the email had been sent via TLS?
Not very many SMTP servers use TLS by default (even if both sides have it available).
-
@Carnival-Boy said:
I don't keep passwords listed in Word documents. I don't think they should either.
In this type of situation, where do you keep the passwords for all of your different clients for all of their different systems?
Personally I don't mind if they use Word/Excel to store these. The best I can hope for is that they are being stored in a safe manor - i.e. not everyone in their company has access to the files, even better if stored on encrypted drives, etc.
