Hyper-V Failover Cluster FAILURE(S)
-
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender the 172.20 addresses are accessible from the 172.30 block.
Why is this necessary?
What network is the DATTO on? -
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@tim_g said in Hyper-V Failover Cluster FAILURE(S):
If the only thing that changed was the IP addressing of (your nodes?), then it may be a DNS related issue. Check all of your cluster related DNS/IP.
eh? You would expect DNS to be in use on the SAN network?
Obviously not, I re-read, and missed that bit about the error being specific to the SAN.
This means there's a communication interruption (most likely) between a node(s) and the CSV, which is on the SAN.
Are there any more related errors or is there any additional error info to go by?
-
@tim_g said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@tim_g said in Hyper-V Failover Cluster FAILURE(S):
If the only thing that changed was the IP addressing of (your nodes?), then it may be a DNS related issue. Check all of your cluster related DNS/IP.
eh? You would expect DNS to be in use on the SAN network?
Obviously not, I re-read, and missed that bit about the error being specific to the SAN.
This means there's a communication interruption (most likely) between a node(s) and the CSV, which is on the SAN.
Are there any more related errors or is there any additional error info to go by?
I'll post the logs when I get back to the house. A typical 5142 and a few others referencing the inability for the VM's not being able to contact the CSV.
-
@tim_g
Error,10/11/2017 1:26:43 PM,Ntfs,55,None,"A corruption was discovered in the file system structure on volume \?\Volume{efbdae12-b4db-11e3-9409-f992ab11bd5e}.
The exact nature of the corruption is unknown. The file system structures need to be scanned and fixed offline.
"
Warning,10/11/2017 1:26:43 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: ??, DeviceName: .
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:43 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: ??, DeviceName: .
({Device Busy}
The device is currently busy.)"
Error,10/11/2017 1:26:43 PM,Ntfs,55,None,"A corruption was discovered in the file system structure on volume \?\Volume{efbdae12-b4db-11e3-9409-f992ab11bd5e}.The exact nature of the corruption is unknown. The file system structures need to be scanned and fixed offline.
"
Warning,10/11/2017 1:26:43 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: ??, DeviceName: .
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:43 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: ??, DeviceName: .
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:43 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: ??, DeviceName: .
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:43 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: Quorum, DeviceName: \Device\HarddiskVolume15.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:43 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: Quorum, DeviceName: \Device\HarddiskVolume15.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:42 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: ??, DeviceName: .
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:42 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: ??, DeviceName: .
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:39 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Information,10/11/2017 1:26:35 PM,Application Popup,26,None,Application popup: Windows - Delayed Write Failed : Exception Processing Message 0xc0000222 Parameters 0x000007FFDA197438 0x000007FFDA197438 0x000007FFDA197438 0x000007FFDA197438
Warning,10/11/2017 1:26:35 PM,Ntfs,50,None,{Delayed Write Failed} Windows was unable to save all the data for the file \Cluster\0.hive. The data has been lost. This error may be caused by a failure of your computer hardware or network connection. Please try to save this file elsewhere.
Warning,10/11/2017 1:26:34 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \?\Volume{efbdae12-b4db-11e3-9409-f992ab11bd5e}, DeviceName: \Device\HarddiskVolume8.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:34 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \?\Volume{efbdae12-b4db-11e3-9409-f992ab11bd5e}, DeviceName: \Device\HarddiskVolume8.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:34 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \?\Volume{efbdae12-b4db-11e3-9409-f992ab11bd5e}, DeviceName: \Device\HarddiskVolume8.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:34 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \?\Volume{efbdae12-b4db-11e3-9409-f992ab11bd5e}, DeviceName: \Device\HarddiskVolume8.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:34 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \?\Volume{efbdae12-b4db-11e3-9409-f992ab11bd5e}, DeviceName: \Device\HarddiskVolume8.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:33 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \?\Volume{efbdae12-b4db-11e3-9409-f992ab11bd5e}, DeviceName: \Device\HarddiskVolume8.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:33 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \?\Volume{efbdae12-b4db-11e3-9409-f992ab11bd5e}, DeviceName: \Device\HarddiskVolume8.
({Device Busy}
The device is currently busy.)"
Error,10/11/2017 1:26:33 PM,Microsoft-Windows-FailoverClustering,1069,Resource Control Manager,"Cluster resource 'Tegile Quorum Drive' of type 'Physical Disk' in clustered role 'Cluster Group' failed.Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it. Check the resource and group state using Failover Cluster Manager or the Get-ClusterResource Windows PowerShell cmdlet."
Error,10/11/2017 1:26:33 PM,Microsoft-Windows-Kernel-General,6,None,An I/O operation initiated by the Registry failed unrecoverably.The Registry could not flush hive (file): ''.
Warning,10/11/2017 1:26:33 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \?\Volume{efbdae12-b4db-11e3-9409-f992ab11bd5e}, DeviceName: \Device\HarddiskVolume8.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:33 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: \?\Volume{efbdae12-b4db-11e3-9409-f992ab11bd5e}, DeviceName: \Device\HarddiskVolume8.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:33 PM,Microsoft-Windows-FailoverClustering,1558,Quorum Manager,The cluster service detected a problem with the witness resource. The witness resource will be failed over to another node within the cluster in an attempt to reestablish access to cluster configuration data.
Error,10/11/2017 1:26:32 PM,Microsoft-Windows-FailoverClustering,1557,Quorum Manager,Cluster service failed to update the cluster configuration data on the witness resource. Please ensure that the witness resource is online and accessible.
Warning,10/11/2017 1:26:30 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:30 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Error,10/11/2017 1:26:29 PM,Microsoft-Windows-FailoverClustering,5142,Cluster Shared Volume,Cluster Shared Volume 'Volume1' ('Tegile CSV3 Volume 3') is no longer accessible from this cluster node because of error 'ERROR_TIMEOUT(1460)'. Please troubleshoot this node's connectivity to the storage device and network connectivity.
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Error,10/11/2017 1:26:29 PM,Microsoft-Windows-FailoverClustering,5142,Cluster Shared Volume,Cluster Shared Volume 'Volume1' ('Tegile CSV3 Volume 3') is no longer accessible from this cluster node because of error 'ERROR_TIMEOUT(1460)'. Please troubleshoot this node's connectivity to the storage device and network connectivity.
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:29 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:28 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:28 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:28 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:28 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:28 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:28 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:28 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:28 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:28 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:28 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:28 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:28 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:28 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:28 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:28 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:27 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:25 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:25 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:25 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:25 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:25 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Warning,10/11/2017 1:26:25 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Error,10/11/2017 1:26:25 PM,Microsoft-Windows-FailoverClustering,5120,Cluster Shared Volume,Cluster Shared Volume 'Volume1' ('Tegile CSV3 Volume 3') is no longer available on this node because of 'STATUS_DEVICE_BUSY(80000011)'. All I/O will temporarily be queued until a path to the volume is reestablished.
Warning,10/11/2017 1:26:25 PM,Microsoft-Windows-Ntfs,140,None,"The system failed to flush data to the transaction log. Corruption may occur in VolumeId: CSV3, DeviceName: \Device\HarddiskVolume14.
({Device Busy}
The device is currently busy.)"
Information,10/11/2017 1:26:25 PM,iScsiPrt,34,None,"A connection to the target was lost, but Initiator successfully reconnected to the target. Dump data contains the target name."
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,7,None,The initiator could not send an iSCSI PDU. Error status is given in the dump data.
Error,10/11/2017 1:26:25 PM,iScsiPrt,20,None,Connection to the target was lost. The initiator will attempt to retry the connection. -
Error,10/11/2017 1:26:32 PM,Microsoft-Windows-FailoverClustering,1557,Quorum Manager,Cluster service failed to update the cluster configuration data on the witness resource. Please ensure that the witness resource is online and accessible.
The Witness uses it's own NIC right? If the Witness isn't working correctly, I'm guessing the switch is the root of the issue, but only a guess.
No logs on the SAN showing anything wrong?
-
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
Error,10/11/2017 1:26:32 PM,Microsoft-Windows-FailoverClustering,1557,Quorum Manager,Cluster service failed to update the cluster configuration data on the witness resource. Please ensure that the witness resource is online and accessible.
The Witness uses it's own NIC right? If the Witness isn't working correctly, I'm guessing the switch is the root of the issue, but only a guess.
No logs on the SAN showing anything wrong?
No issues on the SAN at all. As far as it's concerned everything os working properly.
-
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
Error,10/11/2017 1:26:32 PM,Microsoft-Windows-FailoverClustering,1557,Quorum Manager,Cluster service failed to update the cluster configuration data on the witness resource. Please ensure that the witness resource is online and accessible.
The Witness uses it's own NIC right? If the Witness isn't working correctly, I'm guessing the switch is the root of the issue, but only a guess.
No logs on the SAN showing anything wrong?
I'll have to look and see. I'm not sure as I haven't looked at the Quorum to see how it's configured. With have two hard downs it's wreaked havoc on the DFS and Exchange server.
-
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
Error,10/11/2017 1:26:32 PM,Microsoft-Windows-FailoverClustering,1557,Quorum Manager,Cluster service failed to update the cluster configuration data on the witness resource. Please ensure that the witness resource is online and accessible.
The Witness uses it's own NIC right? If the Witness isn't working correctly, I'm guessing the switch is the root of the issue, but only a guess.
No logs on the SAN showing anything wrong?
No issues on the SAN at all. As far as it's concerned everything os working properly.
Do the logs show traffic coming from both 172.20.200.x and 172.20.201.x to the SAN?
-
@dashrender Actually the Quorum drive is volume 3.
-
My Mistake. iSCSI 2014.3 is actually listed for both the Quorum and the primary LUN that contains all 42 VM's.
-
@scottalanmiller , Is it possible to run local and SAN storage at the same time? Each node has 4 empty slots in each of the 6 nodes.
-
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@scottalanmiller , Is it possible to run local and SAN storage at the same time? Each node has 4 empty slots in each of the 6 nodes.
Yes.
-
What size pipe is between the switch and the router? Can you see if the pipe was tapped out around the time of the failure?
-
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@scottalanmiller , Is it possible to run local and SAN storage at the same time? Each node has 4 empty slots in each of the 6 nodes.
Yes.
Usually I'd load out the servers with tons on storage but they just have 2 drive in each server and they're only 146gb 15K SAS drives ran in RAID1.
-
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@scottalanmiller , Is it possible to run local and SAN storage at the same time? Each node has 4 empty slots in each of the 6 nodes.
Yes.
Usually I'd load out the servers with tons on storage but they just have 2 drive in each server and they're only 146gb 15K SAS drives ran in RAID1.
I'm assuming that's where Hyper-V is installed. FYI, this is a waste of 15K drives - the hypervisor is rarely in use from the drives once it's up and running.
-
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
What size pipe is between the switch and the router? Can you see if the pipe was tapped out around the time of the failure?
It's never tapped out. We keep the Tegile Metrics up on our monitor that shows our SQL server performance and TotalMail server which communicates with the fleet trucks.
-
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@scottalanmiller , Is it possible to run local and SAN storage at the same time? Each node has 4 empty slots in each of the 6 nodes.
Yes.
Usually I'd load out the servers with tons on storage but they just have 2 drive in each server and they're only 146gb 15K SAS drives ran in RAID1.
I'm assuming that's where Hyper-V is installed. FYI, this is a waste of 15K drives - the hypervisor is rarely in use from the drives once it's up and running.
Hyper-V is installed on the server nodes using Server 2012 Datacenter on 2 disks (RAID 1). All VMs are stored in 1 LUN on the SAN.
-
6 hosts, failover cluster with local storage might be challenging, I don't really know anything about it.
I'm sure @scottalanmiller can give some info.
Depending on the age of the hosts, you might find yourself much better off with a two host setup with internal storage and something like StarWinds VSAN. 7 TB internal storage shouldn't be that hard to come by - though the performance needed might require some caching, etc.
-
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@scottalanmiller , Is it possible to run local and SAN storage at the same time? Each node has 4 empty slots in each of the 6 nodes.
Yes.
Usually I'd load out the servers with tons on storage but they just have 2 drive in each server and they're only 146gb 15K SAS drives ran in RAID1.
I'm assuming that's where Hyper-V is installed. FYI, this is a waste of 15K drives - the hypervisor is rarely in use from the drives once it's up and running.
Hyper-V is installed on the server nodes using Server 2012 Datacenter on 2 disks. All VMs are stored in 1 LUN on the SAN.
Sure, this is a typical setup.
-
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@dashrender said in Hyper-V Failover Cluster FAILURE(S):
@kyle said in Hyper-V Failover Cluster FAILURE(S):
@scottalanmiller , Is it possible to run local and SAN storage at the same time? Each node has 4 empty slots in each of the 6 nodes.
Yes.
Usually I'd load out the servers with tons on storage but they just have 2 drive in each server and they're only 146gb 15K SAS drives ran in RAID1.
I'm assuming that's where Hyper-V is installed. FYI, this is a waste of 15K drives - the hypervisor is rarely in use from the drives once it's up and running.
Hyper-V is installed on the server nodes using Server 2012 Datacenter on 2 disks. All VMs are stored in 1 LUN on the SAN.
Sure, this is a typical setup.
I would think with hosting all VM's critical and not so critical in separate LUNS and hell across 2 separate SANs with replication.
We are a 24/7 operation and downtime is a huge no no.
