XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!
-
Can you get to the console of the SAN (PowerVault) to see what it thinks is going on? Is it healthy itself and this is just a XenServer issue... or has storage failed?
-
Given that both hosts are having a problem, and given that the PowerVault is normally the weakest link in the chain and most likely to fail, there is a good chance that that is where the problem lies. That's the shared component. If only one XenServer node was having issues we'd not think that the SAN was the issue. But both at the same time? SAN is the obvious culprit.
-
@scottalanmiller I had gone to another user's machine to log into the PowerVault interface last night. I'm installing the software on my machine now.
-
Also, we're willing to engage someone formally to get this resolved.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller I had gone to another user's machine to log into the PowerVault interface last night. I'm installing the software on my machine now.
That'll likely be the most telling.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
Also, we're willing to engage someone formally to get this resolved.
Let's see if we can't get to the bottom of it quickly, for free first
But that's definitely an option. -
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
Can you get to the console of the SAN (PowerVault) to see what it thinks is going on? Is it healthy itself and this is just a XenServer issue... or has storage failed?
To this question, Xen2 was working fine, and the VMs hosted by Xen2 were working fine for most of yesterday, until I was commanded to shut everything down and bring it back up. It's not just the iSCSI storage that's unplugged. Local storage is also unplugged, as is DVD Drives and Removable Storage. See attached screenshot.
-
Oh okay, that doesn't sound like a SAN problem then.
-
I am logged into the SAN. I can see easily that there's been a virtual disk failure, but that's been there for awhile. I'm not hugely familiar with this interface, but nothing immediately pops out as an issue.
-
Can you tell what has changed between working conditions and now? I know you said that you're at 6.2, but has there been any updates done?
Also, would it be possible to backup the configs for the hosts and reinstall XS from scratch, then restore configs?
-
Also, have you tried shutting both down and bringing up Xen2, since its the pool master? Maybe Xen1 is trying to take over, thinking that Xen2 is down.
-
@NerdyDad said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
Can you tell what has changed between working conditions and now? I know you said that you're at 6.2, but has there been any updates done?
Also, would it be possible to backup the configs for the hosts and reinstall XS from scratch, then restore configs?
@NerdyDad To my knowledge, from the time it was working, last Thursday/Friday, until I walked in the door yesterday morning and found issues, there had been no changes. I'm positive no updates have been done.
With regards to the backups, I'm sure I could try that. Do you have a link handy with a procedure for that, by any chance?
@NerdyDad said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
Also, have you tried shutting both down and bringing up Xen2, since its the pool master? Maybe Xen1 is trying to take over, thinking that Xen2 is down.
That's the last thing I'd tried before posting here. I couldn't bring Xen2 out of maintenance mode, nor reconnect storage.
-
The network issue concerns me the most. That's going to cause everything to fail on its own.
-
Can you ping the SAN from either XS?
-
I think that we need to look at the logs. Do you have access to get onto the machines themselves via SSH?
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@NerdyDad said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
Can you tell what has changed between working conditions and now? I know you said that you're at 6.2, but has there been any updates done?
Also, would it be possible to backup the configs for the hosts and reinstall XS from scratch, then restore configs?
@NerdyDad To my knowledge, from the time it was working, last Thursday/Friday, until I walked in the door yesterday morning and found issues, there had been no changes. I'm positive no updates have been done.
With regards to the backups, I'm sure I could try that. Do you have a link handy with a procedure for that, by any chance?
@NerdyDad said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
Also, have you tried shutting both down and bringing up Xen2, since its the pool master? Maybe Xen1 is trying to take over, thinking that Xen2 is down.
That's the last thing I'd tried before posting here. I couldn't bring Xen2 out of maintenance mode, nor reconnect storage.
-
@Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
Can you ping the SAN from either XS?
I can from Xen2, SAN IPs of .100 and .101 on the mgmt network. I have console control of Xen2 from XenCenter, but Xen1 requires me to run up to the server room. Bear with me.
-
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
I think that we need to look at the logs. Do you have access to get onto the machines themselves via SSH?
I haven't tried yet. Let me get a hold of putty.
-
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
I think that we need to look at the logs. Do you have access to get onto the machines themselves via SSH?
I'm logged into Xen2 with SSH now.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
I think that we need to look at the logs. Do you have access to get onto the machines themselves via SSH?
I'm logged into Xen2 with SSH now.
Great. Let's get as many logs as we can. Likely what we need to know is going to be in there. The logs are under /var/log
