Security mindsets of MSPs



  • About 18 months ago I took out a fairly standard 3rd line support and holiday cover contract with a local MSP to cover for my one-man-band operation when I'm not here. This is a pretty big MSP, employing dozens of people and turning over millions. As part of the contract, an engineer spent a day documenting our systems, and as part of that audit I gave him pretty much all the information someone would need to come in and manage our environment.

    The results of this audit were written up in a Word document and included domain admin passwords, wi-fi/AP passwords, server names and configurations - pretty much everything. The company then e-mailed this document to me, in a completely unsecured e-mail. I e-mailed them back saying I couldn't believe they'd send the domain admin password in an unsecured e-mail and they just replied that they do this for all their clients and I am the first one to ever say anything.

    I had to spend the day changing all of our environment passwords.

    A few years ago, I employed another MSP. This was small, five man operation. Whilst I was away, one of their engineers was providing telephone to support to one of our users and gave the user the domain admin password in order for him to do something. When I returned and found out about it (the user told me), I asked the engineer if he thought that was good practice and was told that it was ok because the user wouldn't remember it.

    I no longer trust external parties with my data. There are too many engineers out there who just don't give a shit about client security, either through laziness or ignorance. Which is a real ball-ache because it makes life harder when I am on holiday. I've created separate domain admin accounts for external support parties, and I keep these accounts disabled until they are needed. But since they are normally only needed when I am not here, enabling them is a problem. They also have their own logmein account running on one of our servers. I understand that cached credentials were vulnerable to Heartbleed, but I don't know if they cached credentials. I suspect they have and haven't bothered to tell me to change passwords (as per Logmein's recent advice).

    I despair. If a Microsoft Gold partner, HP Gold specialist and VMWare Enterprise partner thinks it's normal practice to e-mail Domain Admin and wireless access passwords to clients in an unsecured e-mail, what hope do I have?

    ARRRGGG

    Now I know that most of you are great MSPs, so I'm interested in your best practices. How do you keep client details secure, and how to do you satisfy your clients that you know what you're doing. I still have the problem of how to give external support companies remote administrator access to our servers whilst I'm on holiday. At the moment, I have set-up a domain admin account for a colleague and trained her how to enable and then disable the admin accounts for our external support engineers via AD. I also have the domain admin password printed out and stored in the safe, which about five people here have the key to. I'm working on the premise now to "trust no-one", but still be able to take a holiday.



  • Before talking about how MSPs behave....

    What you need for you concern internally is called a "break glass" system. What that is (I actually just learned this term although obviously not the idea) where an MSP can access a password that you provide (or access a system with a password) that cannot be done without setting off alarms.

    For example, you have a system to which they have access that delivers a sealed password for emergency access. Upon doing so, it is logged, you are alerted, etc. And they have to answer as to why the glass was broken and when they are done, you reset the glass.

    This allows you to provide access for emergencies without granting the access directly. I.e. They get access to get access, they don't just have access.



  • Most MSPs act this way for three reasons:

    1. If you go with primarily "gold" partners, that means they are resellers not MSPs. Their specialty is sales, not IT. They are stores. To get to those levels you don't have to be technical, you have to sell gear. So you bar for what good looks like is actually almost impossible for a quality IT firm to hit but trivial for a non-technical salesman to hit. So you are possibly weeding out the secure firms and looking specifically to the insecure ones.

    2. Most customers train MSPs to behave this way. It's true. Customers not only generally don't care, they often actively demand that MSPs be insecure. That someone wants security is often a massive shock to an MSP.

    3. The MSP should only ever, when possible, have their own passwords. Not always possible, but generally. For example, their own domain admin account just like they were a member of staff. Because you need to know when something is done with your account or with theirs. Then they could only give out their own credentials for which their are liable instead of giving out hours for which they are not.



  • A break glass system (nice term, btw) wouldn't prevent unauthorised access as a result of stolen credentials though. By the time I get the alarm, it is probably too late to do anything.



  • @Carnival-Boy said:

    A break glass system (nice term, btw) wouldn't prevent unauthorised access as a result of stolen credentials though. By the time I get the alarm, it is probably too late to do anything.

    Nothing will prevent that. The idea is to prevent them being stolen.



  • I'm not sure I follow you. What's the purpose of the break-glass system? They still have full access, so it is just as insecure, right?



  • @Carnival-Boy said:

    I'm not sure I follow you. What's the purpose of the break-glass system? They still have full access, so it is just as insecure, right?

    One of the immutable laws of security is that you must always trust your admin(s). Period, no except. Immutable law. Anyone who can act as an admin has unlimited access. That can't be changed.

    So given that, we must do what we can to limit this. There are a few basics:

    1. Audit everything that yo can (if necessary, be warned that auditing is only so useful and can be very counterproductive.)
    2. Be careful whom you trust.
    3. Limit the need to have an admin (break glass, in this example.)

    In the case of the break glass, they only become an admin when necessary. Yes, they could break the glass when not needed, but you'd know about it. There is little alternative.



  • One of the most important documents that any IT pro can know....

    Microsoft's Ten Immutable Laws of Security



  • Law #6: A computer is only as secure as the administrator is trustworthy.



  • I trust them to work on my systems. I don't trust them to prevent credentials from getting into the wrong hands as a result of their sloppy security procedures.

    At the moment they can only come in if I let them, by enabling their AD account. The problem is who let's them in when I'm not here. If my colleague does this, how do I make it easy for my colleague? Alternatively, I could enable their accounts before I go on holiday, and disable them when I return.

    In the days before remote access, they would have to physically enter the building to do any work. There are checks in place to prevent anyone walking in to the building (ie visitor management & security procedures). How do I replicate the physical management procedures in the on-line world? I need a robust system that balances the risk of unauthorised access versus the risk of downtime as a result of no-one being able to access something that needs fixing.



  • @Carnival-Boy said:

    I trust them to work on my systems. I don't trust them to prevent credentials from getting into the wrong hands as a result of their sloppy security procedures.

    You can't differentiate. Distill what you wrote to "I don't trust them."

    So, you can't use them. That's your answer there.



  • @Carnival-Boy said:

    At the moment they can only come in if I let them, by enabling their AD account. The problem is who let's them in when I'm not here. If my colleague does this, how do I make it easy for my colleague? Alternatively, I could enable their accounts before I go on holiday, and disable them when I return.

    Yes. Or have management do it. Turn it on but have management hand over the passwords. Or have management enable/disable and nothing else.



  • @Carnival-Boy said:

    In the days before remote access, they would have to physically enter the building to do any work. There are checks in place to prevent anyone walking in to the building (ie visitor management & security procedures). How do I replicate the physical management procedures in the on-line world? I need a robust system that balances the risk of unauthorised access versus the risk of downtime as a result of no-one being able to access something that needs fixing.

    Well if the building was locked up and they needed to work.... either....

    1. Give them a key and they can get in any time or...
    2. Don't give them a key and trust someone internally with that right to let them in and out.

    Technology doesn't change the choices. It might appear to, but it really doesn't.



  • @scottalanmiller said:

    @Carnival-Boy said:

    I trust them to work on my systems. I don't trust them to prevent credentials from getting into the wrong hands as a result of their sloppy security procedures.

    You can't differentiate. Distill what you wrote to "I don't trust them."

    So, you can't use them. That's your answer there.

    I don't trust anyone external. But I have to take a holiday.



  • @scottalanmiller said:

    Technology doesn't change the choices. It might appear to, but it really doesn't.

    It doesn't change the choices but it changes the solutions. Creating a procedure that allows a colleague to grant admin privileges to external agents isn't a trivial task.



  • Them emailing you the passwords, for lack of a better phrase, is pretty derpy. Next go around, set up separate users and passwords for your main systems. That'll give you the ability to block their access as needed without having to hand over your passwords. If you want to get rid of them, toast their accounts. If you go rogue, they can still help your company keep control.

    Also, take a look at the other types of companies that the MSP works with. Ones that have more security-focused clients will naturally lean towards being more secure.



  • Also realize that email may not be as insecure as you think.

    Before you went off the handle and wasted all that time changing passwords, did you check if the email had been sent via TLS?



  • Went off the handle, LOL. Firstly, I don't consider changing passwords a waste of time. I probably don't change them enough. Secondly, it's the principle of the thing that annoys me. Bad practice is bad practice. I don't keep passwords listed in Word documents. I don't think they should either.



  • @JaredBusch said:

    Also realize that email may not be as insecure as you think.

    Before you went off the handle and wasted all that time changing passwords, did you check if the email had been sent via TLS?

    Not very many SMTP servers use TLS by default (even if both sides have it available).



  • @Carnival-Boy said:

    I don't keep passwords listed in Word documents. I don't think they should either.

    In this type of situation, where do you keep the passwords for all of your different clients for all of their different systems?

    Personally I don't mind if they use Word/Excel to store these. The best I can hope for is that they are being stored in a safe manor - i.e. not everyone in their company has access to the files, even better if stored on encrypted drives, etc.



  • @Dashrender said:

    @JaredBusch said:

    Also realize that email may not be as insecure as you think.

    Before you went off the handle and wasted all that time changing passwords, did you check if the email had been sent via TLS?

    Not very many SMTP servers use TLS by default (even if both sides have it available).

    Office 365 does.



  • This is very interesting to me. As a smaller fish in the IT biz I am trying to do right with security and all of you have a different slant on passwords in emails and in documents. My goal was to password protect my SharePoint OneNote page with my clients User/Pass list. For my web/email hosting clients, I was going to delete all passwords I had on file and require them to create new ones if they forget. I still would have to check how secure my services desks were for sending out passwords for users who request the password reset. Seems like you really can't secure it all, or easily. And how many roadblocks do we want to put in the way of people wanting to get work done?



  • @Carnival-Boy said:

    @scottalanmiller said:

    @Carnival-Boy said:

    I trust them to work on my systems. I don't trust them to prevent credentials from getting into the wrong hands as a result of their sloppy security procedures.

    You can't differentiate. Distill what you wrote to "I don't trust them."

    So, you can't use them. That's your answer there.

    I don't trust anyone external. But I have to take a holiday.

    Then you have a gap. You have to choose. Trust or don't take a holiday.

    What makes the external nature of someone seem distrustful to you but not someone internal? It's the same pool if humans.



  • @Dashrender said:

    @JaredBusch said:

    Also realize that email may not be as insecure as you think.

    Before you went off the handle and wasted all that time changing passwords, did you check if the email had been sent via TLS?

    Not very many SMTP servers use TLS by default (even if both sides have it available).

    Business class ones all do. Maybe those insecure "on premise" people have that problem still but it has been gone for the hosted industry for a long time 🙂



  • @Dashrender said:

    @Carnival-Boy said:

    I don't keep passwords listed in Word documents. I don't think they should either.

    In this type of situation, where do you keep the passwords for all of your different clients for all of their different systems?

    Personally I don't mind if they use Word/Excel to store these. The best I can hope for is that they are being stored in a safe manor - i.e. not everyone in their company has access to the files, even better if stored on encrypted drives, etc.

    Depends on a lot if factors. This one no MSP should disclose in too much detail for obvious security reasons.

    We can say we do a number of things including varying by client, layers of access control, key based access, access control to lists, etc. We have talked about building a custom break glass style system.

    High security passwords are kept in a secure system inside a secure system.

    Highest security are "in brain" only with a physical break glass in case of bus scenario.



  • @scottalanmiller said:

    @Dashrender said:

    @JaredBusch said:

    Also realize that email may not be as insecure as you think.

    Before you went off the handle and wasted all that time changing passwords, did you check if the email had been sent via TLS?

    Not very many SMTP servers use TLS by default (even if both sides have it available).

    Business class ones all do. Maybe those insecure "on premise" people have that problem still but it has been gone for the hosted industry for a long time 🙂

    What do you mean by 'a long time'? Google only went to TLS in light of the revelations of the NSA spying on people. Before that they didn't have TLS enabled by default for SMTP delivery. Hell, they didn't even have encryption between their own datacenters (though.. I will give them a break on that since they believed that those pipes were private and not being spied on through the carrier).



  • One major thing that we do for. Lot if access is limit normal access to coming in through secure central access point and access is then via a key, rather than a password. It's small but it limits problems while making access easier to win at the security vs. usability war.



  • @scottalanmiller said:

    What makes the external nature of someone seem distrustful to you but not someone internal? It's the same pool if humans.

    Nothing at all. It just so happens that there are a couple of people at work that I would trust with my life. I don't have that kind of relationship with any of my external partners. I'd like to and maybe someday I will.

    I'd trust you 🙂 I just don't trust them.

    Another example. Our ERP vendor. Biggish company. Part of Infor. They decide to publish all their support calls on their customer portal as part of a knowledge base. So now I can search other companies support calls to solve my own problems. It's great. But one day, I find a support call that lists the modem number, username and password to remotely access another company's Unix system. They've basically published this information to all their clients around the world.

    My point is, some, but obviously not all, IT companies fail to take their client's security seriously and I think they should know better. This was definitely the case when I was a programmer for a software house - I had no idea about security and no-one ever told me.



  • @Carnival-Boy in that particular case, I would guess that that was an accident. Service providers and vendors do have to be careful as their accident "blast radius" is often larger than in house IT is. But to be fair, we see internal IT people accidentally post rather a bit of stuff when looking for help online or using KB systems that are hosted publicly or whatever.

    Publishing tickets without an air gap process, though, that is pretty idiotic I must admit.


Log in to reply