ZeroTier Site-To-Site



  • ZeroTier Site-To-Site Setup

    ASSUMPTIONS:

    • Site A is on 192.168.10.0/24
    • Site B is on 192.168.122.0/24
    • Site A's VM is 192.168.10.2 for the Local Network
    • Site A's VM is 10.0.0.107 on the ZT Network
    • Site B is 192.168.122.1 on the Local Network
    • Site B is 10.0.0.129 on the ZT Network.

    Step 1: Build a Private Network on https://my.zerotier.com

    992b3cf8-f3e5-4de9-8d4a-60b52db30adb-image.png

    Step 2: Spin up a Linux VM at each site. Connect and authorize them to the ZT Network and note their IP address. For instance:

    Some folks have reported SIGNIFICANT performance improvement when using 2 cores / 2 vcpus for the Linux VMs.
    2975d5f6-c92e-497d-a610-eeee655b39f0-image.png

    Step 2B. Enable IP_Forward:

    Follow your distribution's instructions to enable ip_forward and make it a permanent change... On most distros, this should work:

    sudo echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf

    You can then sysctl -p /etc/sysctl.conf to reload the configuration or reboot.

    sysctl net.ipv4.ip_forward should return

    net.ipv4.ip_forward = 1
    

    if everything is going to work correctly.

    Step 3: From either of the Linux VMs, ensure that they can ping one another on the ZT Subnet.

    Step 4: Set up the Routes inside on https://my.zerotier.com

    0d591b3c-e97d-42c9-84a7-2f3b9b39b467-image.png

    *Once you set up the routes in ZeroTier Central, you do not have to manually add them to your Linux VMs.

    Step 5: Set up the Site Routes at the Routers for Site A and Site B

    SITE A Main Router:
    ae0eaa8c-6b56-4edb-95a3-76a625e8faaa-image.png

    You'll notice for the router at Site A that I am using the INTERNAL network address of my Linux VM.

    SITE A Linux Router VM:

    [email protected] /root # ip route
    default via 192.168.10.1 dev eth0 onlink
    10.0.0.0/24 dev zt1  proto kernel  scope link  src 10.0.0.107
    192.168.10.0/24 dev eth0 proto kernel  scope link  src 192.168.10.2
    192.168.20.0/24 via 10.0.0.116 dev zt1
    192.168.122.0/24 via 10.0.0.129 dev zt1
    

    SITE B, KVM Server, no need for separate VM:

    [email protected]:/root# ip route
    default via <my public ip> dev eth0 onlink
    10.0.0.0/24 dev zt1 scope link  #ZT Subnet
    192.168.10.0/24 via 10.0.0.107 dev zt1 #SiteA, 10.0.0.107 is the ZT IP for the Linux VM at Site A
    192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1  #This server is Site B
    

    **This was done on systems that do not have UFW or firewall-cmd enabled. You may have to set them up to allow traffic between your sites.

    Okay... I think that's it. Mine is working. 🙂

    If you have any questions or comments, ask away!



  • @dafyre Problem with this config : we lose 90% (or more) of the flow. It's not nothing !

    Iperf from a ZT client in Azure to a ZT router on my datacenter (1 GB)
    [ 4] 0.00-10.00 sec 112 MBytes 94.4 Mbits/sec sender
    [ 4] 0.00-10.00 sec 112 MBytes 94.4 Mbits/sec receiver

    Iperf from a ZT client in Azure to a lan machine via the same ZT router my datacenter (1 GB)
    [ 4] 0.00-10.00 sec 112 MBytes 9.5 Mbits/sec sender
    [ 4] 0.00-10.00 sec 112 MBytes 9.4 Mbits/sec receiver

    My ZT routers are Debian on ESX VM

    Or can be the problem?



  • Great write up! Thanks! I looked at ZeroTier a while ago, and because I couldn't get a site-to-site going in under ten minutes, I moved on.

    My only (professional) need for VPNs these days is site to site.

    I'll give it another go and follow your write up.

    Thanks again!



  • @lionelb said in ZeroTier Site-To-Site:

    @dafyre Problem with this config : we lose 90% (or more) of the flow. It's not nothing !

    Iperf from a ZT client in Azure to a ZT router on my datacenter (1 GB)
    [ 4] 0.00-10.00 sec 112 MBytes 94.4 Mbits/sec sender
    [ 4] 0.00-10.00 sec 112 MBytes 94.4 Mbits/sec receiver

    Iperf from a ZT client in Azure to a lan machine via the same ZT router my datacenter (1 GB)
    [ 4] 0.00-10.00 sec 112 MBytes 9.5 Mbits/sec sender
    [ 4] 0.00-10.00 sec 112 MBytes 9.4 Mbits/sec receiver

    My ZT routers are Debian on ESX VM

    Or can be the problem?

    @lionelb said in ZeroTier Site-To-Site:

    @dafyre Problem with this config : we lose 90% (or more) of the flow. It's not nothing !

    Iperf from a ZT client in Azure to a ZT router on my datacenter (1 GB)
    [ 4] 0.00-10.00 sec 112 MBytes 94.4 Mbits/sec sender
    [ 4] 0.00-10.00 sec 112 MBytes 94.4 Mbits/sec receiver

    Iperf from a ZT client in Azure to a lan machine via the same ZT router my datacenter (1 GB)
    [ 4] 0.00-10.00 sec 112 MBytes 9.5 Mbits/sec sender
    [ 4] 0.00-10.00 sec 112 MBytes 9.4 Mbits/sec receiver

    My ZT routers are Debian on ESX VM

    Or can be the problem?

    @lionelb -- What happens if you go from a client in your LAN to a client in your Azure setup?

    You could also try to tracert from the Azure Client to your LAN client and make sure it's not doing anything strange.



  • @JasGot said in ZeroTier Site-To-Site:

    Great write up! Thanks! I looked at ZeroTier a while ago, and because I couldn't get a site-to-site going in under ten minutes, I moved on.

    My only (professional) need for VPNs these days is site to site.

    I'll give it another go and follow your write up.

    Thanks again!

    If you have any questions, do let me know!

    This is the set up that I am still using today.



  • @dafyre I think I found the problem, my VM Debian 9 which serves as router had only 1 GB of RAM and 1 vCPU, I made x 2 and it is much better !
    e86ac6b9-e601-488b-b581-9e2a61069542-image.png
    Here we see the difference before and after ...



  • @lionelb said in ZeroTier Site-To-Site:

    @dafyre I think I found the problem, my VM Debian 9 which serves as router had only 1 GB of RAM and 1 vCPU, I made x 2 and it is much better !
    e86ac6b9-e601-488b-b581-9e2a61069542-image.png
    Here we see the difference before and after ...

    Thanks for the heads up. I made a note of that in Step 2 for making the Linux router VMs.



  • @dafyre And dont use Debian Buster (10), prefear Jessie (8) or Stretch (9) max at this day 🙂



  • @lionelb said in ZeroTier Site-To-Site:

    @dafyre And dont use Debian Buster (10), prefear Jessie (8) or Stretch (9) max at this day 🙂

    I'd recommend always using the current release, so you get security updates and such. Has Buster been released yet?



  • @dafyre said in ZeroTier Site-To-Site:

    @lionelb said in ZeroTier Site-To-Site:

    @dafyre And dont use Debian Buster (10), prefear Jessie (8) or Stretch (9) max at this day 🙂

    I'd recommend always using the current release, so you get security updates and such. Has Buster been released yet?

    Its been out for awhile now.



  • @black3dynamite said in ZeroTier Site-To-Site:

    @dafyre said in ZeroTier Site-To-Site:

    @lionelb said in ZeroTier Site-To-Site:

    @dafyre And dont use Debian Buster (10), prefear Jessie (8) or Stretch (9) max at this day 🙂

    I'd recommend always using the current release, so you get security updates and such. Has Buster been released yet?

    Its been out for awhile now.

    Thanks. The last month or so, I have time to eat, breathe, work, sleep, and help kiddo with his homework, lol... and work has been stupid busy for the last 3 or 4 months and I have no idea why.