Remote management of VMs hosted in colocation
-
@scottalanmiller said in Remote management of VMs hosted in colocation:
@black3dynamite said in Remote management of VMs hosted in colocation:
ZeroTier makes it super easy and safe. If you want, you could make a ZeroTier Bridge VM. I would still setup Fail2Ban and use key-based authentication.
I'd argue that it doesn't really make it safer. ZT is just key based access, like SSH is. If you are concerned that SSH with keys isn't enough, ZT isn't either. ZT exposes even more should it be compromised, as well. In both cases, being compromised is pretty bad, but one is a lot more limited and able to be limited. If you feel someone can breach your keys with SSH, why not ZT just the same?
Not true.
- I never said I don't trust SSH key based auth. I said
@jaredbusch said in Remote management of VMs hosted in colocation:
Keys or not, I will not open SSH to the public internet if I have a simple way around it.
Because, while I may trust SSH, we live in reality where unknown vulnerabilities do exist. By not having it open, at all, to a public facing source, you drastically mitigate the attack surface.
ZT is access without a public inbound allowance. Someone would have to gain access to your ZT network before being able to then try and get your keys for the SSH session.
@scottalanmiller said in Remote management of VMs hosted in colocation:
ZT exposes even more should it be compromised, as well
This is a strawman. If you want to take up this argument, it would be a separate, yet related, discussion.
-
@jaredbusch said in Remote management of VMs hosted in colocation:
ZT is access without a public inbound allowance. Someone would have to gain access to your ZT network before being able to then try and get your keys for the SSH session.
that's my point. How is the access to your ZT network more secure than the keyed access to an SSH port? Both have the risk of vulnerabilities (but nothing has the eyes on it and reviews of OpenSSH, so if that's a concern that makes SSH the clean place to start), and both have access from the outside. Both rely on keys and both have known open ports.
-
@scottalanmiller said in Remote management of VMs hosted in colocation:
@jaredbusch said in Remote management of VMs hosted in colocation:
ZT is access without a public inbound allowance. Someone would have to gain access to your ZT network before being able to then try and get your keys for the SSH session.
that's my point. How is the access to your ZT network more secure than the keyed access to an SSH port? Both have the risk of vulnerabilities (but nothing has the eyes on it and reviews of OpenSSH, so if that's a concern that makes SSH the clean place to start), and both have access from the outside. Both rely on keys and both have known open ports.
I was wondering this exact same thing - why is ZT more trusted in this case than SSH?
-
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
If you're using an infested machine - when you use the apps to make that remote session to the colo, be it SSH or SC, the malware can likely grab the needed information and provide it to the hackers.
I'm guessing that the Salt idea might give the greatest protection here when combined with the IP lock option that Scott mentioned - but then I ask - how do you securely manage the Saltmaster, where ever it lives?
-
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
Umm, no. Exactly the opposite. The point is the reduced attack surface to the host.
@dashrender said in Remote management of VMs hosted in colocation:
I was wondering this exact same thing - why is ZT more trusted in this case than SSH?
@dashrender said in Remote management of VMs hosted in colocation:
but then I ask - how do you securely manage the Saltmaster, where ever it lives?
Because ZT has account level auth (your password to sign in to your account) and then device auth (approving the device that wants to join the network).
Why create all this overhead for no reason, when ZT already handles it?
Answer? Because Salt is SAM's shiny toy.
-
@jaredbusch said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
Umm, no. Exactly the opposite. The point is the reduced attack surface to the host.
@dashrender said in Remote management of VMs hosted in colocation:
I was wondering this exact same thing - why is ZT more trusted in this case than SSH?
@dashrender said in Remote management of VMs hosted in colocation:
but then I ask - how do you securely manage the Saltmaster, where ever it lives?
Because ZT has account level auth (your password to sign in to your account) and then device auth (approving the device that wants to join the network).
Why create all this overhead for no reason, when ZT already handles it?
Answer? Because Salt is SAM's shiny toy.
Pot calling the kettle black there. SSH is the simple, safe, battle tested option. ZT is great and I really like it, but if someone has a toy here, that's it. Yes, it has multiple log in spots, but you just up the key strength to match that and you are at the same security level.
Yes, ZT already handles some of this, but SSH is simple and easy and built in. ZT takes quite a bit more work and is much more complex to do something really simple, it's just not necessary or beneficial when the simple solution handles it.
-
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
The OP's goal is discovering good practices for managing remotely hosted VMs. I identified a potential risk with how I'm doing stuff now as "if my laptop gets compromised, then it's game over for my host, since there is an always-on connection to it via ZT."
-
@scottalanmiller said in Remote management of VMs hosted in colocation:
@jaredbusch said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
Umm, no. Exactly the opposite. The point is the reduced attack surface to the host.
@dashrender said in Remote management of VMs hosted in colocation:
I was wondering this exact same thing - why is ZT more trusted in this case than SSH?
@dashrender said in Remote management of VMs hosted in colocation:
but then I ask - how do you securely manage the Saltmaster, where ever it lives?
Because ZT has account level auth (your password to sign in to your account) and then device auth (approving the device that wants to join the network).
Why create all this overhead for no reason, when ZT already handles it?
Answer? Because Salt is SAM's shiny toy.
Pot calling the kettle black there. SSH is the simple, safe, battle tested option. ZT is great and I really like it, but if someone has a toy here, that's it. Yes, it has multiple log in spots, but you just up the key strength to match that and you are at the same security level.
Yes, ZT already handles some of this, but SSH is simple and easy and built in. ZT takes quite a bit more work and is much more complex to do something really simple, it's just not necessary or beneficial when the simple solution handles it.
More work than your gimmicky Salt setup? As if.
-
Cyber security is just like physical security. Nothing is 100% secure if it's usable.
Isn't the point of security to delay the attacker long enough so he can be detected? That's why we need layers and some kind of intrusion detection.
Don't know anything about it but ZT sounds like one layer of security. Does it at have 2FA?
-
@eddiejennings said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
The OP's goal is discovering good practices for managing remotely hosted VMs. I identified a potential risk with how I'm doing stuff now as "if my laptop gets compromised, then it's game over for my host, since there is an always-on connection to it via ZT."
I'm not sure the always connected bit matters - does it? If you have an SSH client with the key stored in it - and why wouldn't you - this is likely just as bad as ZT. The attacker can just launch the SSH client on your machine and tada - he's got a connection to your host.
The same might be say able about SC - your laptop is compromised - they still your SC creds - hell, they now use your creds from their own machine to connect to SC.
I'm not seeing anything that offers protection from you using a compromised machine.
-
@jaredbusch said in Remote management of VMs hosted in colocation:
@scottalanmiller said in Remote management of VMs hosted in colocation:
@jaredbusch said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
Umm, no. Exactly the opposite. The point is the reduced attack surface to the host.
@dashrender said in Remote management of VMs hosted in colocation:
I was wondering this exact same thing - why is ZT more trusted in this case than SSH?
@dashrender said in Remote management of VMs hosted in colocation:
but then I ask - how do you securely manage the Saltmaster, where ever it lives?
Because ZT has account level auth (your password to sign in to your account) and then device auth (approving the device that wants to join the network).
Why create all this overhead for no reason, when ZT already handles it?
Answer? Because Salt is SAM's shiny toy.
Pot calling the kettle black there. SSH is the simple, safe, battle tested option. ZT is great and I really like it, but if someone has a toy here, that's it. Yes, it has multiple log in spots, but you just up the key strength to match that and you are at the same security level.
Yes, ZT already handles some of this, but SSH is simple and easy and built in. ZT takes quite a bit more work and is much more complex to do something really simple, it's just not necessary or beneficial when the simple solution handles it.
More work than your gimmicky Salt setup? As if.
Well - I think Scott was dumbing it down to simply using SSH, and not using salt to open IPs to allow connections, etc. That would be simpler than using ZT.
-
@dashrender said in Remote management of VMs hosted in colocation:
@eddiejennings said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
The OP's goal is discovering good practices for managing remotely hosted VMs. I identified a potential risk with how I'm doing stuff now as "if my laptop gets compromised, then it's game over for my host, since there is an always-on connection to it via ZT."
I'm not sure the always connected bit matters - does it? If you have an SSH client with the key stored in it - and why wouldn't you - this is likely just as bad as ZT. The attacker can just launch the SSH client on your machine and tada - he's got a connection to your host.
The same might be say able about SC - your laptop is compromised - they still your SC creds - hell, they now use your creds from their own machine to connect to SC.
I'm not seeing anything that offers protection from you using a compromised machine.
You can password protect the key. Or if you use Apps like MobaXTerm and such, they can be set to require a password before the app is usable.
-
@dashrender said in Remote management of VMs hosted in colocation:
@jaredbusch said in Remote management of VMs hosted in colocation:
@scottalanmiller said in Remote management of VMs hosted in colocation:
@jaredbusch said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
Umm, no. Exactly the opposite. The point is the reduced attack surface to the host.
@dashrender said in Remote management of VMs hosted in colocation:
I was wondering this exact same thing - why is ZT more trusted in this case than SSH?
@dashrender said in Remote management of VMs hosted in colocation:
but then I ask - how do you securely manage the Saltmaster, where ever it lives?
Because ZT has account level auth (your password to sign in to your account) and then device auth (approving the device that wants to join the network).
Why create all this overhead for no reason, when ZT already handles it?
Answer? Because Salt is SAM's shiny toy.
Pot calling the kettle black there. SSH is the simple, safe, battle tested option. ZT is great and I really like it, but if someone has a toy here, that's it. Yes, it has multiple log in spots, but you just up the key strength to match that and you are at the same security level.
Yes, ZT already handles some of this, but SSH is simple and easy and built in. ZT takes quite a bit more work and is much more complex to do something really simple, it's just not necessary or beneficial when the simple solution handles it.
More work than your gimmicky Salt setup? As if.
Well - I think Scott was dumbing it down to simply using SSH, and not using salt to open IPs to allow connections, etc. That would be simpler than using ZT.
But that is not what I was arguing. Thus, as I mentioned before, strawman.
-
@pete-s said in Remote management of VMs hosted in colocation:
Cyber security is just like physical security. Nothing is 100% secure if it's usable.
Isn't the point of security to delay the attacker long enough so he can be detected? That's why we need layers and some kind of intrusion detection.
Don't know anything about it but ZT sounds like one layer of security. Does it at have 2FA?
For the management account log in? Not yet.
-
@dafyre said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
@eddiejennings said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
The OP's goal is discovering good practices for managing remotely hosted VMs. I identified a potential risk with how I'm doing stuff now as "if my laptop gets compromised, then it's game over for my host, since there is an always-on connection to it via ZT."
I'm not sure the always connected bit matters - does it? If you have an SSH client with the key stored in it - and why wouldn't you - this is likely just as bad as ZT. The attacker can just launch the SSH client on your machine and tada - he's got a connection to your host.
The same might be say able about SC - your laptop is compromised - they still your SC creds - hell, they now use your creds from their own machine to connect to SC.
I'm not seeing anything that offers protection from you using a compromised machine.
You can password protect the key. Or if you use Apps like MobaXTerm and such, they can be set to require a password before the app is usable.
sure, but if the machine is comp'ed, then there is likely a keylogger there capturing that password too.
-
@dashrender said in Remote management of VMs hosted in colocation:
@jaredbusch said in Remote management of VMs hosted in colocation:
@scottalanmiller said in Remote management of VMs hosted in colocation:
@jaredbusch said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
Umm, no. Exactly the opposite. The point is the reduced attack surface to the host.
@dashrender said in Remote management of VMs hosted in colocation:
I was wondering this exact same thing - why is ZT more trusted in this case than SSH?
@dashrender said in Remote management of VMs hosted in colocation:
but then I ask - how do you securely manage the Saltmaster, where ever it lives?
Because ZT has account level auth (your password to sign in to your account) and then device auth (approving the device that wants to join the network).
Why create all this overhead for no reason, when ZT already handles it?
Answer? Because Salt is SAM's shiny toy.
Pot calling the kettle black there. SSH is the simple, safe, battle tested option. ZT is great and I really like it, but if someone has a toy here, that's it. Yes, it has multiple log in spots, but you just up the key strength to match that and you are at the same security level.
Yes, ZT already handles some of this, but SSH is simple and easy and built in. ZT takes quite a bit more work and is much more complex to do something really simple, it's just not necessary or beneficial when the simple solution handles it.
More work than your gimmicky Salt setup? As if.
Well - I think Scott was dumbing it down to simply using SSH, and not using salt to open IPs to allow connections, etc. That would be simpler than using ZT.
Right, SSH vs. ZT. Basically they have equal security mechanisms. ZT has slightly more risk, because it exposes slightly more. SSH is vastly simpler to implement in most cases, and is built in.
Again, I love ZT, it's just overkill for something basic. It's a common myth that VPNs are super secure and other things are not, SSH is a VPN itself, and all the username/password/key combinations are really just "longer key length" at the end of the day, they don't represent additional security.
-
@pete-s said in Remote management of VMs hosted in colocation:
Don't know anything about it but ZT sounds like one layer of security. Does it at have 2FA?
ZT is a software defined network built using a mix of hub and spoke, and mesh VPN tech. It's very advanced and very good. If you use it purely in a point to point like mode, then you have quite a bit of security, but you still need SSH or whatever at that point. The extra layer, if it is truly completely extra, provides more security by doubling up on the channel, but it also makes things more complex. If you use it for any more points than just the one to one, it becomes a point of exposure above and beyond what was there before.
-
@jaredbusch said in Remote management of VMs hosted in colocation:
@scottalanmiller said in Remote management of VMs hosted in colocation:
@jaredbusch said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
Umm, no. Exactly the opposite. The point is the reduced attack surface to the host.
@dashrender said in Remote management of VMs hosted in colocation:
I was wondering this exact same thing - why is ZT more trusted in this case than SSH?
@dashrender said in Remote management of VMs hosted in colocation:
but then I ask - how do you securely manage the Saltmaster, where ever it lives?
Because ZT has account level auth (your password to sign in to your account) and then device auth (approving the device that wants to join the network).
Why create all this overhead for no reason, when ZT already handles it?
Answer? Because Salt is SAM's shiny toy.
Pot calling the kettle black there. SSH is the simple, safe, battle tested option. ZT is great and I really like it, but if someone has a toy here, that's it. Yes, it has multiple log in spots, but you just up the key strength to match that and you are at the same security level.
Yes, ZT already handles some of this, but SSH is simple and easy and built in. ZT takes quite a bit more work and is much more complex to do something really simple, it's just not necessary or beneficial when the simple solution handles it.
More work than your gimmicky Salt setup? As if.
I'm not talking about Salt, that was to answer a very specific question that would STILL need to be addressed with your ZT setup. But not relevant here.
I was talking about how much simpler SSH was than your gimmicky ZT setup. SSH isn't some pet technology here. And with your ZT, how are you connecting if you aren't using SSH, RDP or something afterwards anyway? It's an extra step.
-
@scottalanmiller said in Remote management of VMs hosted in colocation:
@eddiejennings said in Remote management of VMs hosted in colocation:
Allowing an SSH connection to the managementVM from the Internet
I have not tried this approach yet, and it appears more risky than the Screen Connect approach, since SSH to that VM would be open to the Internet. Unless I'm missing some benefit to this approach, I'll not be using it.
Use a strong key, lock to your IP. Very safe. Add Fail2Ban, of course.
Or add Salt and open/close based on need so it doesn't stay open.
Fail2ban doesn't work with keys.
-
@eddiejennings said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
The OP's goal is discovering good practices for managing remotely hosted VMs. I identified a potential risk with how I'm doing stuff now as "if my laptop gets compromised, then it's game over for my host, since there is an always-on connection to it via ZT."
If you have SSH or ZT configured to "just connect" on your laptop, and the laptop is compromised, you have a problem. That's a given. Neither forces you to do that. In either case you could make that connection be on demand and make it so that the machine itself never has enough information to connect on its own. So what you mention as a risk is real, but can also be mitigated - it's not a risk implied by the use of either solution.
The risk in either case is also that without your laptop, someone might guess the keys. Not likely, making it a small risk, but both ZT and SSH are vulnerable to someone without your laptop brute forcing the key or finding a vulnerability in the code and breaking through. This is where I feel SSH wins hands down, ZT is great, but OpenSSH is famous for being one of the most solid, reliable, and audited pieces of security code ever written. It's made by the OpenBSD team and their commitment to security is legendary. It's supported by every big name in the industry, including Microsoft (and IBM, Red Hat, Canonical / Ubuntu, Suse, Oracle, Apple, and on and on) and is pretty much impossible to beat when it comes to the code being looked at.
Both approaches are super secure. The two aren't exact rip and replace, there are some assumptions we have to get around... like does ZT assume only exposure of a jump box, or all boxes? Does SSH mean direct to all, or only to a jump box? Does ZT expose a box that is limited to only SSH anyway? Does the box with the first SSH hop have access to anything other than SSH? And so forth.
SSH is simpler than ZT. No third party tools needed. ZT isn't complex, but it's not "just click a button", either. You have to configure the networking and sometimes with the IPv6 it doesn't like to communicate by default. ZT requires you to have ZT installed on any box you want to access from, which isn't extra security as it does nothing to stop bad guys attacking it, but does make it harder for you when you need to access the system and no longer have the device you intended to log in from.
SSH makes things like malware attacks against known vulnerabilities basically not exist. ZT would, if used normally, expose that heavily.
