Upon further scrutiny of the Event Logs, I think I might be on to what is happening. I think it may be a clock issue.
We are an XCP-ng shop, and it seems that the XenServer guest agent is attempting to sync the clock every 30 minutes. It issues a sync which causes the time to shift anywhere from a few milliseconds to a few MINUTES.
Seems this is an issue other folks have complained about (though others seem to experience much wider variations in time).
The two jobs that were "skipped" on the 01/31 are scheduled to run at 7:00 PM. It looks like the system time was tweaked at 6:39 PM and 7:09 PM. So, in theory, the jobs should've fired, but this is definitely not helping things. These are events with the ID of 1, the "source" is Kernel-General, and the user is SYSTEM.
01/31 @ 6:39:18 PM:
The system time has changed to 2020-02-01T02:39:18.826000000Z from 2020-02-01T02:40:37.936724600Z.
01/31 @ 7:09:17 PM:
The system time has changed to 2020-02-01T03:09:17.907000000Z from 2020-02-01T03:10:33.882678100Z.
This eventually causes the Time Service to complain since it causes the time to be off by more than 5 seconds for a period of time.
Every time it seems to roll the clock back. I followed the guidance via the Citrix forum post linked above in modifying registry entries to stop it from doing this but it didn't seem to help. Supposedly modifying the following registry entries to be either "blank" or set to "Local" is supposed to stop the Xen agent time sync.
However, they were already blank in the registry. So I set them to "Local" and restarted all the services on the box that have Citrix or Xen in the name since I can't reboot this box without pre-planning (yay for 24/7 availability). I'm still seeing the Kernel-General Event ID 1's in the System event log. Fun...
The time on the hosts are correct, and my NTP hierarchy is as so:
Member Servers -> DCs -> Stratum 1 NTP servers hosted by Hurricane Electric (one in San Jose, CA and the other in New York, NY).
I combed through time stuff yesterday and the time on the DCs seems stable (and w32tm reports they are syncing properly), as well as on other hosts, but I'll need to give it a second look.
I am also planning on upgrading from XCP-ng 7.6 to 8.0 this weekend...I'll have to see if this behavior continues after that.