@dafyre said in VMQ issues/Veeam/Windows Server 2019...:

@Jimmy9008 said in VMQ issues/Veeam/Windows Server 2019...:

@dafyre said in VMQ issues/Veeam/Windows Server 2019...:

@Jimmy9008 said in VMQ issues/Veeam/Windows Server 2019...:

@dafyre said in VMQ issues/Veeam/Windows Server 2019...:

@Jimmy9008 said in VMQ issues/Veeam/Windows Server 2019...:

Hi folks,

I am having VMQ issues on my 3 x Windows Server 2019 host. I am using a team of Intel X550 NIC (3 x 10 GbE interfaces to 3 x Stacked switch). The Team has been configured using Switch Independent mode.

Now, VMs are on the LAN perfectly and communicate. Rock solid.

Now to the issue. When I use Veeam with Application Aware turned on, the hosts get stuck creating checkpoints for VMs at 9%. The only option is to kill the host, restart and turn the VMs back on. It happens every time. (Strangely, checkpoints run fine native).

Veeam have looked at the logs and have said its an issue they have seen before, and to turn off VMQ initially on NICs and VMs as that often solves the issue. No worries. I do that, and the backups then run fine - perfect!

However, VMs over time then drop off of the network. I can connect to them in Hyper-V, but nothing I do will bring them back on the network. Initially they are on the network, just at some point in time many drop off, whilst others stay on.

The only resolution is to turn VMQ back on and reboot. I cant really keep testing this either as it causes much downtime! Not good. Of course, when VMQ is back on... Application Aware backups then fail and kill the host like I said at the start

Any idea why some VMs drop off of the network with VMQ disabled on the VM and NIC? Host, fully patched. NIC, latest firmware. I thought you didnt have to use VMQ...

Best,
Jim

Is the the same VMs that drop off the network every time?

What OS are the VMs that get dropped off the network?

What kind of switches are you running?

1. Uncertain. I've noticed a few are the same, but I've had no time to really look in to it. Too many users. I'm trying to get ideas that could be the issue to do proper testing with downtime around Dec 20th when a lot are on holiday.

2. I think 2012 R2. Possibly 2016 too. Can't recall any 2019.

3. Dell N4064 Stack.

Try the ArpRetryCount trick. You'll have to reboot each VM or host after you make the change.

Ok, I can try that. So:

1. disable VMQ physical
2. disable VMQ on all VM NIC
3. change registry
4. reboot

Sound about right?

3. change the registry on any machine that is down.

But yeah, that sounds about right.

Do I need to change the reg on the host too?

@dafyre said in VMQ issues/Veeam/Windows Server 2019...:

@Jimmy9008 said in VMQ issues/Veeam/Windows Server 2019...:

@dafyre said in VMQ issues/Veeam/Windows Server 2019...:

@Jimmy9008 said in VMQ issues/Veeam/Windows Server 2019...:

Hi folks,

I am having VMQ issues on my 3 x Windows Server 2019 host. I am using a team of Intel X550 NIC (3 x 10 GbE interfaces to 3 x Stacked switch). The Team has been configured using Switch Independent mode.

Now, VMs are on the LAN perfectly and communicate. Rock solid.

Now to the issue. When I use Veeam with Application Aware turned on, the hosts get stuck creating checkpoints for VMs at 9%. The only option is to kill the host, restart and turn the VMs back on. It happens every time. (Strangely, checkpoints run fine native).

Veeam have looked at the logs and have said its an issue they have seen before, and to turn off VMQ initially on NICs and VMs as that often solves the issue. No worries. I do that, and the backups then run fine - perfect!

However, VMs over time then drop off of the network. I can connect to them in Hyper-V, but nothing I do will bring them back on the network. Initially they are on the network, just at some point in time many drop off, whilst others stay on.

The only resolution is to turn VMQ back on and reboot. I cant really keep testing this either as it causes much downtime! Not good. Of course, when VMQ is back on... Application Aware backups then fail and kill the host like I said at the start

Any idea why some VMs drop off of the network with VMQ disabled on the VM and NIC? Host, fully patched. NIC, latest firmware. I thought you didnt have to use VMQ...

Best,
Jim

Is the the same VMs that drop off the network every time?

What OS are the VMs that get dropped off the network?

What kind of switches are you running?

For Windows systems (Even sometimes a Physical Server, I usually have to do this in regedit:

I haven't run into anything like that with our Linux systems. You may need to do it at the Host level on HyperV as well.

That's something I can try in December. So do that to the VMs and hosts?

Yeah. Or you can try it the next time a VM goes down. That's what we did. Make the change, reboot the VM, and it's usually happy again.

That's the thing. The VM are rock solid with VMQ and Veeam, just only when no App Aware backups are ran.

@dafyre said in VMQ issues/Veeam/Windows Server 2019...:

@Jimmy9008 said in VMQ issues/Veeam/Windows Server 2019...:

@dafyre said in VMQ issues/Veeam/Windows Server 2019...:

@Jimmy9008 said in VMQ issues/Veeam/Windows Server 2019...:

Hi folks,

I am having VMQ issues on my 3 x Windows Server 2019 host. I am using a team of Intel X550 NIC (3 x 10 GbE interfaces to 3 x Stacked switch). The Team has been configured using Switch Independent mode.

Now, VMs are on the LAN perfectly and communicate. Rock solid.

Now to the issue. When I use Veeam with Application Aware turned on, the hosts get stuck creating checkpoints for VMs at 9%. The only option is to kill the host, restart and turn the VMs back on. It happens every time. (Strangely, checkpoints run fine native).

Veeam have looked at the logs and have said its an issue they have seen before, and to turn off VMQ initially on NICs and VMs as that often solves the issue. No worries. I do that, and the backups then run fine - perfect!

However, VMs over time then drop off of the network. I can connect to them in Hyper-V, but nothing I do will bring them back on the network. Initially they are on the network, just at some point in time many drop off, whilst others stay on.

The only resolution is to turn VMQ back on and reboot. I cant really keep testing this either as it causes much downtime! Not good. Of course, when VMQ is back on... Application Aware backups then fail and kill the host like I said at the start

Any idea why some VMs drop off of the network with VMQ disabled on the VM and NIC? Host, fully patched. NIC, latest firmware. I thought you didnt have to use VMQ...

Best,
Jim

Is the the same VMs that drop off the network every time?

What OS are the VMs that get dropped off the network?

What kind of switches are you running?

1. Uncertain. I've noticed a few are the same, but I've had no time to really look in to it. Too many users. I'm trying to get ideas that could be the issue to do proper testing with downtime around Dec 20th when a lot are on holiday.

2. I think 2012 R2. Possibly 2016 too. Can't recall any 2019.

3. Dell N4064 Stack.

Try the ArpRetryCount trick. You'll have to reboot each VM or host after you make the change.

Ok, I can try that. So:

1. disable VMQ physical
2. disable VMQ on all VM NIC
3. change registry
4. reboot

Sound about right?

@dafyre said in VMQ issues/Veeam/Windows Server 2019...:

@Jimmy9008 said in VMQ issues/Veeam/Windows Server 2019...:

Hi folks,

I am having VMQ issues on my 3 x Windows Server 2019 host. I am using a team of Intel X550 NIC (3 x 10 GbE interfaces to 3 x Stacked switch). The Team has been configured using Switch Independent mode.

Now, VMs are on the LAN perfectly and communicate. Rock solid.

Now to the issue. When I use Veeam with Application Aware turned on, the hosts get stuck creating checkpoints for VMs at 9%. The only option is to kill the host, restart and turn the VMs back on. It happens every time. (Strangely, checkpoints run fine native).

Veeam have looked at the logs and have said its an issue they have seen before, and to turn off VMQ initially on NICs and VMs as that often solves the issue. No worries. I do that, and the backups then run fine - perfect!

However, VMs over time then drop off of the network. I can connect to them in Hyper-V, but nothing I do will bring them back on the network. Initially they are on the network, just at some point in time many drop off, whilst others stay on.

The only resolution is to turn VMQ back on and reboot. I cant really keep testing this either as it causes much downtime! Not good. Of course, when VMQ is back on... Application Aware backups then fail and kill the host like I said at the start

Any idea why some VMs drop off of the network with VMQ disabled on the VM and NIC? Host, fully patched. NIC, latest firmware. I thought you didnt have to use VMQ...

Best,
Jim

Is the the same VMs that drop off the network every time?

What OS are the VMs that get dropped off the network?

What kind of switches are you running?

For Windows systems (Even sometimes a Physical Server, I usually have to do this in regedit:

I haven't run into anything like that with our Linux systems. You may need to do it at the Host level on HyperV as well.

That's something I can try in December. So do that to the VMs and hosts?

@dafyre said in VMQ issues/Veeam/Windows Server 2019...:

@Jimmy9008 said in VMQ issues/Veeam/Windows Server 2019...:

Hi folks,

I am having VMQ issues on my 3 x Windows Server 2019 host. I am using a team of Intel X550 NIC (3 x 10 GbE interfaces to 3 x Stacked switch). The Team has been configured using Switch Independent mode.

Now, VMs are on the LAN perfectly and communicate. Rock solid.

Now to the issue. When I use Veeam with Application Aware turned on, the hosts get stuck creating checkpoints for VMs at 9%. The only option is to kill the host, restart and turn the VMs back on. It happens every time. (Strangely, checkpoints run fine native).

Veeam have looked at the logs and have said its an issue they have seen before, and to turn off VMQ initially on NICs and VMs as that often solves the issue. No worries. I do that, and the backups then run fine - perfect!

However, VMs over time then drop off of the network. I can connect to them in Hyper-V, but nothing I do will bring them back on the network. Initially they are on the network, just at some point in time many drop off, whilst others stay on.

The only resolution is to turn VMQ back on and reboot. I cant really keep testing this either as it causes much downtime! Not good. Of course, when VMQ is back on... Application Aware backups then fail and kill the host like I said at the start

Any idea why some VMs drop off of the network with VMQ disabled on the VM and NIC? Host, fully patched. NIC, latest firmware. I thought you didnt have to use VMQ...

Best,
Jim

Is the the same VMs that drop off the network every time?

What OS are the VMs that get dropped off the network?

What kind of switches are you running?

1. Uncertain. I've noticed a few are the same, but I've had no time to really look in to it. Too many users. I'm trying to get ideas that could be the issue to do proper testing with downtime around Dec 20th when a lot are on holiday.

2. I think 2012 R2. Possibly 2016 too. Can't recall any 2019.

3. Dell N4064 Stack.

@Dashrender said in Veeam Active Full or Synthetic Full?:

@DustinB3403 said in Veeam Active Full or Synthetic Full?:

& Replication accesses the previous full backup file and a chain of subsequent incremental backup files on the backup

Reverse it on him - ask him how active full is safer?

If you don't trust the data in the incrementals and the previous full backup, then why are you using the product?

Putting load on production shortens its lifespan - maybe not noticeably, but it still does because of work being done.

I like this idea. That could work well - thank you.

Are they safer? If not, why do active full even exist?

@DustinB3403 said in Veeam Active Full or Synthetic Full?:

@Jimmy9008 said in Veeam Active Full or Synthetic Full?:

@DustinB3403 said in Veeam Active Full or Synthetic Full?:

So to your coworkers point, I would say, no there is no need for a separate full backup weekly.

He is daying npt to use synthetics on a saturday. Just do the incremental and an active full

I'm going to take some guesses fixing this.

He is saying not to use synthetics on Saturday, just use incrementals and a separate active full

Is that correct?

Yes.

@DustinB3403 said in Veeam Active Full or Synthetic Full?:

So to your coworkers point, I would say, no there is no need for a separate full backup weekly.

He is daying npt to use synthetics on a saturday. Just do the incremental and an active full

@DustinB3403 Yeah, I read that already but it dodnt help clarify on those questions I had.