Co-lo + 5 (or more) sites....connect 'em all
-
@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:
@scottalanmiller said in Co-lo + 5 (or more) sites....connect 'em all:
@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:
@Dashrender said in Co-lo + 5 (or more) sites....connect 'em all:
@Pete-S said in Co-lo + 5 (or more) sites....connect 'em all:
@JaredBusch said in Co-lo + 5 (or more) sites....connect 'em all:
@Dashrender said in Co-lo + 5 (or more) sites....connect 'em all:
@Aaron-Studer said in Co-lo + 5 (or more) sites....connect 'em all:
My question is why? Why setup ZT instead of site to site on all the devices?
I suppose one answer could be, because it's just a single setup, instead of 5 setups.
WTF?
FFS, the question is about connecting multiple colo's. Do you only have one thing in each colo? Most don't. The OP specifically mentioned multiple thigns.
You smokin?
"The co-lo has all the gear (servers, voip, apps, file shares etc).
You have 5 (or more) sites that "connect" to the co-lo."What we aren't told - is there a firewall in front of all of that stuff at the co-lo, or is it all directly on the internet? Then the OP asks - can ZT be installed on ER? I'll admit I was assuming an ER at each location, and at the co-lo in front of all of that gear.
Yes, the plan is an ER in front at all locations (that plan isn't set in stone)
We did this for a company from their colo but NOT with ZT, ERs using their native, much faster IPSec.
Did you use Route based VPN?
https://help.ubnt.com/hc/en-us/articles/115011377588-EdgeRouter-IPsec-Route-Based-VTI-Site-to-Site-VPNI've done both. No idea on speed difference. never ran in to router limits with both methods.
-
@JaredBusch said in Co-lo + 5 (or more) sites....connect 'em all:
@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:
@scottalanmiller said in Co-lo + 5 (or more) sites....connect 'em all:
@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:
@Dashrender said in Co-lo + 5 (or more) sites....connect 'em all:
@Pete-S said in Co-lo + 5 (or more) sites....connect 'em all:
@JaredBusch said in Co-lo + 5 (or more) sites....connect 'em all:
@Dashrender said in Co-lo + 5 (or more) sites....connect 'em all:
@Aaron-Studer said in Co-lo + 5 (or more) sites....connect 'em all:
My question is why? Why setup ZT instead of site to site on all the devices?
I suppose one answer could be, because it's just a single setup, instead of 5 setups.
WTF?
FFS, the question is about connecting multiple colo's. Do you only have one thing in each colo? Most don't. The OP specifically mentioned multiple thigns.
You smokin?
"The co-lo has all the gear (servers, voip, apps, file shares etc).
You have 5 (or more) sites that "connect" to the co-lo."What we aren't told - is there a firewall in front of all of that stuff at the co-lo, or is it all directly on the internet? Then the OP asks - can ZT be installed on ER? I'll admit I was assuming an ER at each location, and at the co-lo in front of all of that gear.
Yes, the plan is an ER in front at all locations (that plan isn't set in stone)
We did this for a company from their colo but NOT with ZT, ERs using their native, much faster IPSec.
Did you use Route based VPN?
https://help.ubnt.com/hc/en-us/articles/115011377588-EdgeRouter-IPsec-Route-Based-VTI-Site-to-Site-VPNI've done both. No idea on speed difference. never ran in to router limits with both methods.
Ease of setup/ability to add more sites, one method vs the other?
-
@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:
@dafyre said in Co-lo + 5 (or more) sites....connect 'em all:
I'm up to 3 sites for the moment. Once of them goes away in about 2 weeks.
I connect them all via ZeroTier.
How's the speeds between sites?
Speeds were good. I don't remember what they were, but I transferred 1TB of stuff over ZT without any issues.
-
@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:
@dafyre said in Co-lo + 5 (or more) sites....connect 'em all:
I'm up to 3 sites for the moment. Once of them goes away in about 2 weeks.
I connect them all via ZeroTier.
This is you: https://mangolassi.it/topic/19493/zerotier-site-to-site
How has it worked out so far?Yeah, that's me, and it's been great. I haven't had any problems with it at all.
-
@scottalanmiller said in Co-lo + 5 (or more) sites....connect 'em all:
@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:
@Dashrender said in Co-lo + 5 (or more) sites....connect 'em all:
@Pete-S said in Co-lo + 5 (or more) sites....connect 'em all:
@JaredBusch said in Co-lo + 5 (or more) sites....connect 'em all:
@Dashrender said in Co-lo + 5 (or more) sites....connect 'em all:
@Aaron-Studer said in Co-lo + 5 (or more) sites....connect 'em all:
My question is why? Why setup ZT instead of site to site on all the devices?
I suppose one answer could be, because it's just a single setup, instead of 5 setups.
WTF?
FFS, the question is about connecting multiple colo's. Do you only have one thing in each colo? Most don't. The OP specifically mentioned multiple thigns.
You smokin?
"The co-lo has all the gear (servers, voip, apps, file shares etc).
You have 5 (or more) sites that "connect" to the co-lo."What we aren't told - is there a firewall in front of all of that stuff at the co-lo, or is it all directly on the internet? Then the OP asks - can ZT be installed on ER? I'll admit I was assuming an ER at each location, and at the co-lo in front of all of that gear.
Yes, the plan is an ER in front at all locations (that plan isn't set in stone)
We did this for a company from their colo but NOT with ZT, ERs using their native, much faster IPSec.
Which is what i was mentioning up top.
-
@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:
@JaredBusch said in Co-lo + 5 (or more) sites....connect 'em all:
@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:
@scottalanmiller said in Co-lo + 5 (or more) sites....connect 'em all:
@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:
@Dashrender said in Co-lo + 5 (or more) sites....connect 'em all:
@Pete-S said in Co-lo + 5 (or more) sites....connect 'em all:
@JaredBusch said in Co-lo + 5 (or more) sites....connect 'em all:
@Dashrender said in Co-lo + 5 (or more) sites....connect 'em all:
@Aaron-Studer said in Co-lo + 5 (or more) sites....connect 'em all:
My question is why? Why setup ZT instead of site to site on all the devices?
I suppose one answer could be, because it's just a single setup, instead of 5 setups.
WTF?
FFS, the question is about connecting multiple colo's. Do you only have one thing in each colo? Most don't. The OP specifically mentioned multiple thigns.
You smokin?
"The co-lo has all the gear (servers, voip, apps, file shares etc).
You have 5 (or more) sites that "connect" to the co-lo."What we aren't told - is there a firewall in front of all of that stuff at the co-lo, or is it all directly on the internet? Then the OP asks - can ZT be installed on ER? I'll admit I was assuming an ER at each location, and at the co-lo in front of all of that gear.
Yes, the plan is an ER in front at all locations (that plan isn't set in stone)
We did this for a company from their colo but NOT with ZT, ERs using their native, much faster IPSec.
Did you use Route based VPN?
https://help.ubnt.com/hc/en-us/articles/115011377588-EdgeRouter-IPsec-Route-Based-VTI-Site-to-Site-VPNI've done both. No idea on speed difference. never ran in to router limits with both methods.
Ease of setup/ability to add more sites, one method vs the other?
Well, once you have ZT setup, adding another site is likely the easiest. You just add ZT on a new ER, join the mesh and you're done.
With site to site VPN, you'd have to build the tunnel on both ER's (the co-lo and the new site). Not that this is hard, just possible a tiny more amount of work.
-
@Dashrender said in Co-lo + 5 (or more) sites....connect 'em all:
Well, once you have ZT setup, adding another site is likely the easiest. You just add ZT on a new ER, join the mesh and you're done.
Who has done this ZT on ER install?
The previous blog post seems to imply heavy/high CPU usage, wondering how this would affect performance? -
@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:
The previous blog post seems to imply heavy/high CPU usage, wondering how this would affect performance?
We'd expect a bit. OpenVPN does as it is. SSL VPNs take a toll on performance.
-
@scottalanmiller said in Co-lo + 5 (or more) sites....connect 'em all:
@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:
The previous blog post seems to imply heavy/high CPU usage, wondering how this would affect performance?
We'd expect a bit. OpenVPN does as it is. SSL VPNs take a toll on performance.
It's not OpenVPN that takes a toll on performance. If you look at the actual overhead on the packets it's very small.
But it's the fact that small routers have very weak CPUs but they can off load straight IPsec, when you are not doing packet inspection or anything that requires the CPU. However they can't off load OpenVPN.
If you look at more powerful CPUs, like Intel, you can off load OpenVPN with the AES-NI extensions in the CPU. So OpenVPN barely makes a dent on the CPU if you run it over a WAN link.
-
@Pete-S said in Co-lo + 5 (or more) sites....connect 'em all:
@scottalanmiller said in Co-lo + 5 (or more) sites....connect 'em all:
@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:
The previous blog post seems to imply heavy/high CPU usage, wondering how this would affect performance?
We'd expect a bit. OpenVPN does as it is. SSL VPNs take a toll on performance.
It's not OpenVPN that takes a toll on performance. If you look at the actual overhead on the packets it's very small.
But it's the fact that small routers have very weak CPUs but they can off load straight IPsec, when you are not doing packet inspection or anything that requires the CPU. However they can't off load OpenVPN.
If you look at more powerful CPUs, like Intel, you can off load OpenVPN with the AES-NI extensions in the CPU. So OpenVPN barely makes a dent on the CPU if you run it over a WAN link.
PS. So high CPU is not linked to the protocol but to what the router support for hardware off load.
-
@Pete-S said in Co-lo + 5 (or more) sites....connect 'em all:
But it's the fact that small routers have very weak CPUs but they can off load straight IPsec, when you are not doing packet inspection or anything that requires the CPU. However they can't off load OpenVPN.
Sounds like the choice should def be IPSec for less of a performance hit?
-
@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:
@Pete-S said in Co-lo + 5 (or more) sites....connect 'em all:
But it's the fact that small routers have very weak CPUs but they can off load straight IPsec, when you are not doing packet inspection or anything that requires the CPU. However they can't off load OpenVPN.
Sounds like the choice should def be IPSec for less of a performance hit?
With an Edgerouter yes. You can read more here and see how much difference it makes.
https://help.ubnt.com/hc/en-us/articles/115006567467-EdgeRouter-Hardware-Offloading -
@Pete-S said in Co-lo + 5 (or more) sites....connect 'em all:
@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:
@Pete-S said in Co-lo + 5 (or more) sites....connect 'em all:
But it's the fact that small routers have very weak CPUs but they can off load straight IPsec, when you are not doing packet inspection or anything that requires the CPU. However they can't off load OpenVPN.
Sounds like the choice should def be IPSec for less of a performance hit?
With an Edgerouter yes. You can read more here and see how much difference it makes.
https://help.ubnt.com/hc/en-us/articles/115006567467-EdgeRouter-Hardware-OffloadingAlso note that even with IPsec it's very dependent on what encryption you are using.
AES-256-GCM for instance would kill the Edgerouter performance but coast on a x86 server with AES-NI (which every CPUs has except some low powered devices).. -
@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:
@Pete-S said in Co-lo + 5 (or more) sites....connect 'em all:
But it's the fact that small routers have very weak CPUs but they can off load straight IPsec, when you are not doing packet inspection or anything that requires the CPU. However they can't off load OpenVPN.
Sounds like the choice should def be IPSec for less of a performance hit?
Pretty much always. That's why IPSec is the de facto protocol for normal VPN usage, to the point that people confuse other things like ZT or OpenVPN as "alternatives" rather than all of them being peers. Every major VPN platform uses IPsec because it is built in to nearly everything and is extremely light to implement.
-
Here are some benchmarks on IPsec with some different edgerouters.
https://www.simonmott.co.uk/2018/08/ubiquiti-edgerouter-ipsec-performance/
From the link it says the more powerful ER-4 will top out at about 450 Mbps of IPsec using AES-128. -
Hmmm...is this an option...? https://www.tnsr.com/
-
@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:
Hmmm...is this an option...? https://www.tnsr.com/
An option in general? Sure, it's just a vRouter that does IPsec. I'm sure it is good, but you can't run it on an EdgeRouter because it's an OS.
-
@scottalanmiller said in Co-lo + 5 (or more) sites....connect 'em all:
@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:
Hmmm...is this an option...? https://www.tnsr.com/
An option in general? Sure, it's just a vRouter that does IPsec. I'm sure it is good, but you can't run it on an EdgeRouter because it's an OS.
One would have to switch to pfSense if TNSR is a viable option.
-
@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:
@scottalanmiller said in Co-lo + 5 (or more) sites....connect 'em all:
@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:
Hmmm...is this an option...? https://www.tnsr.com/
An option in general? Sure, it's just a vRouter that does IPsec. I'm sure it is good, but you can't run it on an EdgeRouter because it's an OS.
One would have to switch to pfSense if TNSR is a viable option.
I guess the real question I'd have is... why? What about TNSR makes it interesting in any way? Aren't you just looking at replacing tried and true, built in IPSec implementations with this complicated package that is just repacking OpenSwan?
I'm confused what you are trying to achieve. Connecting 5+ sites is the absolute clear use case for normal everyday IPSec on your outside hardware router. This is as "by the textbook" as it gets.
Can you use other VPN tech for this like OpenVPN, yes. Should you? Not really, it has no benefits to you. IPSec is best for this for speed, support, ease of use.
This is not a case where ZT has applicability unless you have needs that haven't been mentioned. Same with TNSR, what would this do other than make simple IPSec really hard and complicated for no reason?
This feels like one of those Aaron threads where he's captivated by all kinds of shiny product pages and misses that he's trying to do something very straightforward that is handled best by the tools that everyone uses for this every day. I'm missing what is driving the attempt to research new, hip, flashy products as none of them seem to bring anything to this particular table.
-
@FATeknollogee said in Co-lo + 5 (or more) sites....connect 'em all:
What options are available today?
VPN, ZeroTier??Keep in mind anything that does this is a VPN. ZT and others are not VPN alternatives, they are just VPNs. VPN (or leased lines) are the only possible options at the end of the day.
