Routing problem - can this be done?



  • I was wondering if this can be solved somehow.

    Customer has a bunch of devices that have static IPs. They all belong to the same subnet and the IP address is the unique identifier.

    Now, we want to add a number of 4G routers. Each devices can at any time be connected either to the LAN or to one of the 4G routers. How can this be setup so the traffic goes to the right device?

    routing_problem.png



  • BTW, there is no communication between the devices themselves. Only from the servers in the datacenter to the devices.



  • Is adding all of the devices and the servers in the data centre to a ZeroTier network an option? You wouldn't even need to set up a traditional S2S VPN for that.



  • Zerotier makes sense here. I've also deployed a Sophos SG Appliance (yes I know) at a small private datacenter and use their RED (Remote Ethernet Device) units behind cellular modems for branch offices on construction sites. Work extremely well and no issues. Use them for data and voip.



  • Zerotier is not possible to use in this case because the devices have fixed firmware and the routers don't support ZT.

    But I'm thinking maybe L2TP tunnels is a way forward. L2 is Layer 2 but I don't know if that means that it will work like a switch and send anything regardless of IP address or anything else at Layer 3.



  • @Pete-S said in Routing problem - can this be done?:

    But I'm thinking maybe L2TP tunnels is a way forward. L2 is Layer 2 but I don't know if that means that it will work like a switch and send anything regardless of IP address or anything else at Layer 3.

    Should mean like a switch, yes. But a switch knows where the MACs are, so it's not indiscriminate.



  • @Pete-S said in Routing problem - can this be done?:

    Zerotier is not possible to use in this case because the devices have fixed firmware and the routers don't support ZT.

    That sucks.



  • Could you use one of those travel routers as a go-between the 4G and the device? I just picked up one of these to play with a USB 4G dongle for a road trip I have to take. It supports OpenWRT and VPN connections/terminations:
    https://www.gl-inet.com/products/gl-ar750s/
    Or is it the 4G is built into the device and no way to separate it?



  • @jt1001001 said in Routing problem - can this be done?:

    Or is it the 4G is built into the device and no way to separate it?

    No, 4G router is separate but the router already has support for IPSEC, OpenVPN etc so I think it's fine.

    It's the static IP addressing of the devices that messes things up and the fact that the customer has to be able to put any device on any 4G router or on the LAN. That means that all the 4G routers must have the same subnet config.

    So makes me think that what we really want is to stretch layer 2 to all these devices. My research on that has led me to L2TP, GRE and VXLAN.

    • I don't know if L2TP can do the job. L2TPv3 supposedly can.
    • I don't know if GRE can do the job either. I know NVGRE can in a vritaul environment but don't know the difference.
    • VXLAN should also work but the devices don't support it.

    At this point I feel like I don't know enough about networking and routing because then I would know what would work and what wouldn't.



  • If the network at the endpoints is the same single subnet, how does this router
    1cfe3e3f-cdf0-4dc8-9f3c-aa96f84ed0a2-image.png

    know which VPN to push the traffic to? The need for Layer 2 transmission so that all the VPN tunnels get the traffic seems like a must - which means a ton of traffic not meant for those lines would be pushed... unless there is a way to have the above router know specifically which IP is on which VPN - but I don't know how you would do that - assuming you even can?



  • @Dashrender said in Routing problem - can this be done?:

    If the network at the endpoints is the same single subnet, how does this router
    1cfe3e3f-cdf0-4dc8-9f3c-aa96f84ed0a2-image.png

    know which VPN to push the traffic to? The need for Layer 2 transmission so that all the VPN tunnels get the traffic seems like a must - which means a ton of traffic not meant for those lines would be pushed... unless there is a way to have the above router know specifically which IP is on which VPN - but I don't know how you would do that - assuming you even can?

    Good question - I don't know how. With one VPN link to the LAN everything is simple... Just add another router and everything gets super complicated.



  • @Pete-S said in Routing problem - can this be done?:

    @Dashrender said in Routing problem - can this be done?:

    If the network at the endpoints is the same single subnet, how does this router
    1cfe3e3f-cdf0-4dc8-9f3c-aa96f84ed0a2-image.png

    know which VPN to push the traffic to? The need for Layer 2 transmission so that all the VPN tunnels get the traffic seems like a must - which means a ton of traffic not meant for those lines would be pushed... unless there is a way to have the above router know specifically which IP is on which VPN - but I don't know how you would do that - assuming you even can?

    Good question - I don't know how. With one VPN link to the LAN everything is simple... Just add another router and everything gets super complicated.

    Of course - the problem is that you're splitting a layer 2 network over multiple connections - it's not meant to work like that - sure something can sit in the middle and fake that setup out.. but it adds complication for sure.

    Can we ask why all the devices must be on the same layer 2 network?



  • @Dashrender said in Routing problem - can this be done?:

    Can we ask why all the devices must be on the same layer 2 network?

    They don't have to be per se. It's just that they have to have static IPs since that is how the servers know which one is which. Right now all of them are connected to the LAN on their own VLAN and in their own subnet. There are more devices than on the drawing, about 50 or so.

    The suggestion is that when they have to move them around the end user will disconnect the LAN cable and connect it to a 4G router instead. And everything just magically works.

    Static DHCP will not work in this application since the devices are replaced and serviced and rotated by local tech 24/7. The techs can change IP address on the device but are not allowed and not capable of administering the DHCP servers.



  • wow - that kinda sucks!

    I'm just spitballing here.

    What about putting each one on it's own VLAN, Then they can still have their assigned static IP and routing can work.

    OK perhaps not, because as long as they are on the LAN, there has to be a route at the master router in your list that tells packets how to route, but if the device changes to a 4G connection, it will suddenly move to it's own VPN tunnel - and you'll be right where you are with the current dilemma, needing to split the L2 between two different VPN tunnels.



  • @Pete-S said in Routing problem - can this be done?:

    They don't have to be per se. It's just that they have to have static IPs since that is how the servers know which one is which.

    Clearly here is an example that shows why coding to an IP is a horrible practice. Coding to a hostname would be much better - Best would be some code/number/whatever assigned to the device that exists outside of it's communication method.



  • @Dashrender said in Routing problem - can this be done?:

    @Pete-S said in Routing problem - can this be done?:

    They don't have to be per se. It's just that they have to have static IPs since that is how the servers know which one is which.

    Clearly here is an example that shows why coding to an IP is a horrible practice. Coding to a hostname would be much better - Best would be some code/number/whatever assigned to the device that exists outside of it's communication method.

    It's possible to change the server to using hostnames but it will cause other problems instead.

    If you define the hosts in the DNS server you still have to use static IPs or you have to use DHCP reservations and set the hostname there and have the DHCP server update the DNS. But if you use DHCP reservations you have to tie the IP to the MAC address which makes the devices not field replaceable.



  • I may have found a solution and that is to skip routing and bridge instead.

    I'll bridge together all VPN connections in the data center router. And use OpenVPN in tap mode (= L2 bridge) for the VPN links. Then all devices can share the same subnet. Broadcast traffic goes everywhere and everyone knows where packets should be sent to reach its destination.

    Hmm, I have to mull it over.



  • @Pete-S said in Routing problem - can this be done?:

    @Dashrender said in Routing problem - can this be done?:

    @Pete-S said in Routing problem - can this be done?:

    They don't have to be per se. It's just that they have to have static IPs since that is how the servers know which one is which.

    Clearly here is an example that shows why coding to an IP is a horrible practice. Coding to a hostname would be much better - Best would be some code/number/whatever assigned to the device that exists outside of it's communication method.

    It's possible to change the server to using hostnames but it will cause other problems instead.

    If you define the hosts in the DNS server you still have to use static IPs or you have to use DHCP reservations and set the hostname there and have the DHCP server update the DNS. But if you use DHCP reservations you have to tie the IP to the MAC address which makes the devices not field replaceable.

    I take it the server reaches out to the devices, not the other way around?

    Ubiquiti APs work by the APs reaching out to the server, not the other way, this means the server doesn't really care what the IP address of the AP is, or what it's name is, etc. When you adopt the AP into the server, it's now a known device. The AP checks in with enough frequency that any updates you make at the server are distributed to the APs at a pace admins are happy with.



  • @Pete-S said in Routing problem - can this be done?:

    But if you use DHCP reservations you have to tie the IP to the MAC address which makes the devices not field replaceable.

    Still field replaceable. In one case the field tech has to set the IP on the device, in the other case their set it on the DHCP server. Same change. But one is more flexible and some changes can be done without a field tech.



  • @scottalanmiller said in Routing problem - can this be done?:

    @Pete-S said in Routing problem - can this be done?:

    But if you use DHCP reservations you have to tie the IP to the MAC address which makes the devices not field replaceable.

    Still field replaceable. In one case the field tech has to set the IP on the device, in the other case their set it on the DHCP server. Same change. But one is more flexible and some changes can be done without a field tech.

    But he already said the field tech doesn't have the ability to change DHCP, so that makes change-outs harder as it requires more people to be involved.



  • @Dashrender said in Routing problem - can this be done?:

    @scottalanmiller said in Routing problem - can this be done?:

    @Pete-S said in Routing problem - can this be done?:

    But if you use DHCP reservations you have to tie the IP to the MAC address which makes the devices not field replaceable.

    Still field replaceable. In one case the field tech has to set the IP on the device, in the other case their set it on the DHCP server. Same change. But one is more flexible and some changes can be done without a field tech.

    But he already said the field tech doesn't have the ability to change DHCP, so that makes change-outs harder as it requires more people to be involved.

    Ah, missed that. Field tech COULD change the MAC address, though.



  • @scottalanmiller said in Routing problem - can this be done?:

    @Dashrender said in Routing problem - can this be done?:

    @scottalanmiller said in Routing problem - can this be done?:

    @Pete-S said in Routing problem - can this be done?:

    But if you use DHCP reservations you have to tie the IP to the MAC address which makes the devices not field replaceable.

    Still field replaceable. In one case the field tech has to set the IP on the device, in the other case their set it on the DHCP server. Same change. But one is more flexible and some changes can be done without a field tech.

    But he already said the field tech doesn't have the ability to change DHCP, so that makes change-outs harder as it requires more people to be involved.

    Ah, missed that. Field tech COULD change the MAC address, though.

    Field techs are technical people but not sysadmins.
    The customer has a 24/7 operation so the field techs are on call and will come in within an hour or so and replace stuff in the middle of the night, on weekends, on Christmas Day or whenever it's needed.



  • @Dashrender said in Routing problem - can this be done?:

    I take it the server reaches out to the devices, not the other way around?

    Yes, that's correct. To use the right terminology it's actually the devices that are the servers and the "servers" in the data center are the clients.



  • Anyway, I will go with bridging for now.

    I don't think bridging will cause any problems in this case. Traffic is intermittent and low speed so even if there are more broadcast traffic sent over the VPN links, compared to a routing solution, I don't think it will have any impact.

    But I'll probably set up some kind of test to make sure before deploying.


Log in to reply