Does intra-VM traffic leave the host?
-
@Pete-S said in Does intra-VM traffic leave the host?:
Assume you have two VMs on the same host talking to each other, on a network that is not just an internal network inside the host.
Will the traffic actually leave the physical NIC on the host or will it just stay inside the host?
From my understanding of ESXi, two guests located on the same local host vswitch will not go out the physical NIC.
-
@pmoncho said in Does intra-VM traffic leave the host?:
@Pete-S said in Does intra-VM traffic leave the host?:
Assume you have two VMs on the same host talking to each other, on a network that is not just an internal network inside the host.
Will the traffic actually leave the physical NIC on the host or will it just stay inside the host?
From my understanding of ESXi, two guests located on the same local host vswitch will not go out the physical NIC.
I wonder if that holds true for KVM and Xen as well?
-
@Pete-S said in Does intra-VM traffic leave the host?:
@pmoncho said in Does intra-VM traffic leave the host?:
@Pete-S said in Does intra-VM traffic leave the host?:
Assume you have two VMs on the same host talking to each other, on a network that is not just an internal network inside the host.
Will the traffic actually leave the physical NIC on the host or will it just stay inside the host?
From my understanding of ESXi, two guests located on the same local host vswitch will not go out the physical NIC.
I wonder if that holds true for KVM and Xen as well?
KVM just uses the hosts network adapters, so of course you can easily setup a host-only network adapter to assign to any guest. Xen should be able to do the same thing, but I forget if it self-manages any networking.
-
@travisdh1 said in Does intra-VM traffic leave the host?:
@Pete-S said in Does intra-VM traffic leave the host?:
@pmoncho said in Does intra-VM traffic leave the host?:
@Pete-S said in Does intra-VM traffic leave the host?:
Assume you have two VMs on the same host talking to each other, on a network that is not just an internal network inside the host.
Will the traffic actually leave the physical NIC on the host or will it just stay inside the host?
From my understanding of ESXi, two guests located on the same local host vswitch will not go out the physical NIC.
I wonder if that holds true for KVM and Xen as well?
KVM just uses the hosts network adapters, so of course you can easily setup a host-only network adapter to assign to any guest. Xen should be able to do the same thing, but I forget if it self-manages any networking.
I'm not after a host only network. I'm wondering if traffic will go out the physical switch on a "regular" external network, if both VMs are on the same host.
What I want to do is set ACLs in the hardware switch to limit intra-VM traffic. But that would only work if the traffic actually goes out to the hardware switch.
-
1.) No
2.) Why would you want separate virtualization platforms on the same subnet, anyway? Managing traffic by subnet is MUCH easier than trying to manage it by host. Flat networks are frowned upon these days because they are much more difficult to manage traffic.
-
@Pete-S said in Does intra-VM traffic leave the host?:
@travisdh1 said in Does intra-VM traffic leave the host?:
@Pete-S said in Does intra-VM traffic leave the host?:
@pmoncho said in Does intra-VM traffic leave the host?:
@Pete-S said in Does intra-VM traffic leave the host?:
Assume you have two VMs on the same host talking to each other, on a network that is not just an internal network inside the host.
Will the traffic actually leave the physical NIC on the host or will it just stay inside the host?
From my understanding of ESXi, two guests located on the same local host vswitch will not go out the physical NIC.
I wonder if that holds true for KVM and Xen as well?
KVM just uses the hosts network adapters, so of course you can easily setup a host-only network adapter to assign to any guest. Xen should be able to do the same thing, but I forget if it self-manages any networking.
I'm not after a host only network. I'm wondering if traffic will go out the physical switch on a "regular" external network, if both VMs are on the same host.
What I want to do is set ACLs in the hardware switch to limit intra-VM traffic. But that would only work if the traffic actually goes out to the hardware switch.
So you want to use your switch as a firewall between your VMs?
Are the VMs on the same subnet?
-
@IRJ said in Does intra-VM traffic leave the host?:
2.) Why would you want separate virtualization platforms on the same subnet, anyway? Managing traffic by subnet is MUCH easier than trying to manage it by host. Flat networks are frowned upon these days because they are much more difficult to manage traffic.
I'm increasing the segmentation granularity so the traffic is already managed by subnet, however there are a few subnets I would like to lock down even more and it's just too much to make a subnet for every single host.
-
@Dashrender said in Does intra-VM traffic leave the host?:
So you want to use your switch as a firewall between your VMs?
Are the VMs on the same subnet?Yes, more or less. Firewall can only filter between subnets. And yes, VMs are in the same subnet.
-
@Pete-S said in Does intra-VM traffic leave the host?:
@Dashrender said in Does intra-VM traffic leave the host?:
So you want to use your switch as a firewall between your VMs?
Are the VMs on the same subnet?Yes, more or less. Firewall can only filter between subnets. And yes, VMs are in the same subnet.
Interesting - so your switch can do ACL at what? the MAC level? If not, how is it applying an ACL between machines that are on the same subnet? I suppose it could be IP based, but what about non IP based protocols?
I'm sure JB will blast me for lack of understanding - I'm asking because I'm unaware of the ability to do this..
-
@Dashrender said in Does intra-VM traffic leave the host?:
@Pete-S said in Does intra-VM traffic leave the host?:
@Dashrender said in Does intra-VM traffic leave the host?:
So you want to use your switch as a firewall between your VMs?
Are the VMs on the same subnet?Yes, more or less. Firewall can only filter between subnets. And yes, VMs are in the same subnet.
Interesting - so your switch can do ACL at what? the MAC level? If not, how is it applying an ACL between machines that are on the same subnet? I suppose it could be IP based, but what about non IP based protocols?
I'm sure JB will blast me for lack of understanding - I'm asking because I'm unaware of the ability to do this..
If we are talking physical devices connected to ports on a switch then yes. You can set access control based on MAC address (L2) or IP address (L3).
-
Using Open vSwitch with KVM or Xen, you can do some advanced switching like traffic filtering.
https://www.openvswitch.org/ -
@black3dynamite said in Does intra-VM traffic leave the host?:
Using Open vSwitch with KVM or Xen, you can do some advanced switching like traffic filtering.
https://www.openvswitch.org/Hmm, I running xcp-ng on our hosts and if I understand correctly it uses openvswitch internally. But I have no clue how you can send commands to the switch. I read that Citrix has some VM that runs a GUI interface as well but it has been deprecated AFAIK.
-
@Pete-S said in Does intra-VM traffic leave the host?:
@IRJ said in Does intra-VM traffic leave the host?:
2.) Why would you want separate virtualization platforms on the same subnet, anyway? Managing traffic by subnet is MUCH easier than trying to manage it by host. Flat networks are frowned upon these days because they are much more difficult to manage traffic.
I'm increasing the segmentation granularity so the traffic is already managed by subnet, however there are a few subnets I would like to lock down even more and it's just too much to make a subnet for every single host.
Yeah in this case you will want to use a host based firewall. In AWS or Azure you would use Security Groups since they are stateful and at EC2 level. Here you will want to use
iptablesor windows firewall to whitelist ONLY traffic needed. -
@Pete-S said in Does intra-VM traffic leave the host?:
@travisdh1 said in Does intra-VM traffic leave the host?:
@Pete-S said in Does intra-VM traffic leave the host?:
@pmoncho said in Does intra-VM traffic leave the host?:
@Pete-S said in Does intra-VM traffic leave the host?:
Assume you have two VMs on the same host talking to each other, on a network that is not just an internal network inside the host.
Will the traffic actually leave the physical NIC on the host or will it just stay inside the host?
From my understanding of ESXi, two guests located on the same local host vswitch will not go out the physical NIC.
I wonder if that holds true for KVM and Xen as well?
KVM just uses the hosts network adapters, so of course you can easily setup a host-only network adapter to assign to any guest. Xen should be able to do the same thing, but I forget if it self-manages any networking.
I'm not after a host only network. I'm wondering if traffic will go out the physical switch on a "regular" external network, if both VMs are on the same host.
What I want to do is set ACLs in the hardware switch to limit intra-VM traffic. But that would only work if the traffic actually goes out to the hardware switch.
Idk what you're running but with KVM it depends on the type of interface you set up. If you're using the default macvtap interface type it has three modes. VEPA will traverse out to the switch and back but the switch has to support hairpin. Bridge does not, it's direct between guests. There is also private which is like VEPA but won't support hairpin at all.
Why not just set up your internal networks for the guests?
Or like @IRJ said just use the host firewall.
-
Ah nevermind. I just saw you said xen so ignore the KVM info.
-
@IRJ said in Does intra-VM traffic leave the host?:
@Pete-S said in Does intra-VM traffic leave the host?:
@IRJ said in Does intra-VM traffic leave the host?:
2.) Why would you want separate virtualization platforms on the same subnet, anyway? Managing traffic by subnet is MUCH easier than trying to manage it by host. Flat networks are frowned upon these days because they are much more difficult to manage traffic.
I'm increasing the segmentation granularity so the traffic is already managed by subnet, however there are a few subnets I would like to lock down even more and it's just too much to make a subnet for every single host.
Yeah in this case you will want to use a host based firewall. In AWS or Azure you would use Security Groups since they are stateful and at EC2 level. Here you will want to use
iptablesor windows firewall to whitelist ONLY traffic needed.That seems to be the easiest solution to manage.
@black3dynamite I just had a look at openvswitch and it looks very capable. So much so that I think it's too advanced for my simple needs.
-
@Pete-S said in Does intra-VM traffic leave the host?:
Assume you have two VMs on the same host talking to each other, on a network that is not just an internal network inside the host.
Will the traffic actually leave the physical NIC on the host or will it just stay inside the host?
Technically this question is completely dependent on both the products you are using and their configuration. On average, no, this would be a terrible waste of resources and kill performance. But you can certainly make products do this.
-
@Pete-S said in Does intra-VM traffic leave the host?:
What I want to do is set ACLs in the hardware switch to limit intra-VM traffic. But that would only work if the traffic actually goes out to the hardware switch.
That's a terrible design, think of what a huge bottleneck that would be. The ACL idea is great, but do that on the virtual switch instead.
-
@scottalanmiller said in Does intra-VM traffic leave the host?:
@Pete-S said in Does intra-VM traffic leave the host?:
What I want to do is set ACLs in the hardware switch to limit intra-VM traffic. But that would only work if the traffic actually goes out to the hardware switch.
That's a terrible design, think of what a huge bottleneck that would be. The ACL idea is great, but do that on the virtual switch instead.
I had a look at doing just that with openvswitch (OVS) since that is installed by default in xcp-ng/xenserver and can also be installed on KVM.
But it looks to be pretty complicated to accomplish compared to a normal hardware switch. It seems to be a part of OpenFlow / SDN in openvswitch.
-
@Pete-S said in Does intra-VM traffic leave the host?:
@scottalanmiller said in Does intra-VM traffic leave the host?:
@Pete-S said in Does intra-VM traffic leave the host?:
What I want to do is set ACLs in the hardware switch to limit intra-VM traffic. But that would only work if the traffic actually goes out to the hardware switch.
That's a terrible design, think of what a huge bottleneck that would be. The ACL idea is great, but do that on the virtual switch instead.
I had a look at doing just that with openvswitch (OVS) since that is installed by default in xcp-ng/xenserver and can also be installed on KVM.
But it looks to be pretty complicated to accomplish compared to a normal hardware switch. It seems to be a part of OpenFlow / SDN in openvswitch.
Far more common to see it there. It's pretty rare that people want L3 switching and ACLs inside of the backplane. It makes sense that you might want it, it's just pretty niche. I think most people opt to isolate workloads that they need talking onto a single backplane, and those that they want isolated on another. Or they just opt for ACLs on the individual hosts rather than the switch layer to handle that since they control the entire stack, it's not like an external physical device that they might not control.
