XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!
-
Much healthier.
-
We still need to see the logs
-
You might get some useful info from the XenCenter. In one of the earlier posts you had a Screencap of your XC. There were 5700+ notifications listed. I imagine if you click on that and goto Errors and Alerts buttons you will get some useful info. Alerts can be useful for finding the cause, Errors get a bit spammy in XC though they are useful as well.
Generally, the local storage not being connected is probably the first i would look at. That is bad. Then once that is working you need to figure out why your multipathd setup is borked. You said you changed some ip addresses; this is probably a major reason why it isnt working.
Have you tried logging into the switch this is all connected to and seeing if it is ok? -
@momurda said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
You might get some useful info from the XenCenter. In one of the earlier posts you had a Screencap of your XC. There were 5700+ notifications listed. I imagine if you click on that and goto Errors and Alerts buttons you will get some useful info. Alerts can be useful for finding the cause, Errors get a bit spammy in XC though they are useful as well.
Generally, the local storage not being connected is probably the first i would look at. That is bad. Then once that is working you need to figure out why your multipathd setup is borked. You said you changed some ip addresses; this is probably a major reason why it isnt working.
Have you tried logging into the switch this is all connected to and seeing if it is ok?Multipath is likely failing because the network is having issues.
-
What is the network setup on these servers and the SAN?
One switch, multiple switches? how many connections to each server? etc.
-
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
We still need to see the logs
[root@xen2 ~]# tail -n 100 messages tail: cannot open `messages' for reading: No such file or directory -
@momurda said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
You might get some useful info from the XenCenter. In one of the earlier posts you had a Screencap of your XC. There were 5700+ notifications listed. I imagine if you click on that and goto Errors and Alerts buttons you will get some useful info. Alerts can be useful for finding the cause, Errors get a bit spammy in XC though they are useful as well.
Generally, the local storage not being connected is probably the first i would look at. That is bad. Then once that is working you need to figure out why your multipathd setup is borked. You said you changed some ip addresses; this is probably a major reason why it isnt working.
Have you tried logging into the switch this is all connected to and seeing if it is ok?It's a few thousand of these.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
We still need to see the logs
[root@xen2 ~]# tail -n 100 messages tail: cannot open `messages' for reading: No such file or directorymight have to cd to the logs folder and try that tail again.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
It's a few thousand of these.
What about the 5 errors under Events?
-
@Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
What is the network setup on these servers and the SAN?
One switch, multiple switches? how many connections to each server? etc.
2 switches. I tried reaching out to the guy that set it up 6 years ago, and he said he doesn't even remember how he did it. It was the last time he did a XenServer and PowerVault setup. I can get you the networking info from the XC and from the PV, and maybe we can find info for the switch that way, but I have no documentation for it.
In my defense, I've been lobbying for a replacement for this setup. Hopefully, it's coming due now.
-
@Danp said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
It's a few thousand of these.
What about the 5 errors under Events?
All work I've been trying to do since it went down.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
We still need to see the logs
[root@xen2 ~]# tail -n 100 messages tail: cannot open `messages' for reading: No such file or directoryThat's not good. That means that something really went wrong. Probably the disk had filled up.
-
@Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
What is the network setup on these servers and the SAN?
One switch, multiple switches? how many connections to each server? etc.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
What is the network setup on these servers and the SAN?
One switch, multiple switches? how many connections to each server? etc.
2 switches. I tried reaching out to the guy that set it up 6 years ago, and he said he doesn't even remember how he did it. It was the last time he did a XenServer and PowerVault setup. I can get you the networking info from the XC and from the PV, and maybe we can find info for the switch that way, but I have no documentation for it.
In my defense, I've been lobbying for a replacement for this setup. Hopefully, it's coming due now.
XenServer is very good, but the version you have is old and not maintained. Having it with the PowerVault was never acceptable. That was a bad setup. XenServer is a very good choice. But if you don't have a XenServer team, you'll want something else. Of course, @scale is a good choice, but you already know that
Moving this to a Scale cluster would be a quick fix that would address both the compute and the storage needs and let you move to a far more reliable setup. -
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
What is the network setup on these servers and the SAN?
One switch, multiple switches? how many connections to each server? etc.
2 switches. I tried reaching out to the guy that set it up 6 years ago, and he said he doesn't even remember how he did it. It was the last time he did a XenServer and PowerVault setup. I can get you the networking info from the XC and from the PV, and maybe we can find info for the switch that way, but I have no documentation for it.
In my defense, I've been lobbying for a replacement for this setup. Hopefully, it's coming due now.
XenServer is very good, but the version you have is old and not maintained. Having it with the PowerVault was never acceptable. That was a bad setup. XenServer is a very good choice. But if you don't have a XenServer team, you'll want something else. Of course, @scale is a good choice, but you already know that
Moving this to a Scale cluster would be a quick fix that would address both the compute and the storage needs and let you move to a far more reliable setup.Is it possible to somehow spin up what we have on a Scale cluster as is?
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
What is the network setup on these servers and the SAN?
One switch, multiple switches? how many connections to each server? etc.
2 switches. I tried reaching out to the guy that set it up 6 years ago, and he said he doesn't even remember how he did it. It was the last time he did a XenServer and PowerVault setup. I can get you the networking info from the XC and from the PV, and maybe we can find info for the switch that way, but I have no documentation for it.
In my defense, I've been lobbying for a replacement for this setup. Hopefully, it's coming due now.
XenServer is very good, but the version you have is old and not maintained. Having it with the PowerVault was never acceptable. That was a bad setup. XenServer is a very good choice. But if you don't have a XenServer team, you'll want something else. Of course, @scale is a good choice, but you already know that
Moving this to a Scale cluster would be a quick fix that would address both the compute and the storage needs and let you move to a far more reliable setup.Is it possible to somehow spin up what we have on a Scale cluster as is?
Not quite, we need a means of grabbing the files off of it. So we either need a utility to do some conversion which I think Scale can help with; or we need to do a P2V process which requires us to get this working (even if just limping) in the mean time.
-
The iSCSI host ports on the SAN are 192.168.130.101-104. No gateway, no reference to a switch that I can find. I'm hoping a guy that's at lunch has some documentation to help out with that.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
The iSCSI host ports on the SAN are 192.168.130.101-104. No gateway, no reference to a switch that I can find. I'm hoping a guy that's at lunch has some documentation to help out with that.
I dont think that that is where we are stuck, so that's good info to have but not in our critical path at the moment (I don't think.)
-
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
We still need to see the logs
[root@xen2 ~]# tail -n 100 messages tail: cannot open `messages' for reading: No such file or directoryThat's not good. That means that something really went wrong. Probably the disk had filled up.
Is there another way to view log entries?
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
We still need to see the logs
[root@xen2 ~]# tail -n 100 messages tail: cannot open `messages' for reading: No such file or directoryThat's not good. That means that something really went wrong. Probably the disk had filled up.
Is there another way to view log entries?
Not if they are not there. That's not a log viewer, that's the logs itself. Are you use that you used the cd command to get into the right directory before trying to view the logs?
