XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!
-
@Danp said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@Dashrender WinSCP would be handy here for copying the file.
As @Danp said, you can download WinSCP on your workstation, then log into the server using your credentials, then browse to the directory, and drag copy the files to your local computer. Once you've done that you can delete the logs - BUT - wait for Scott before doing any of that - he is an expert, I'm less than a noob.
-
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
Everything that ends in .gz can be safely just moved off (deleting from the source) as they are archival and not being used.
rm /var/log/*.gz
Is that correct? -
@Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@Danp said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@Dashrender WinSCP would be handy here for copying the file.
As @Danp said, you can download WinSCP on your workstation, then log into the server using your credentials, then browse to the directory, and drag copy the files to your local computer. Once you've done that you can delete the logs - BUT - wait for Scott before doing any of that - he is an expert, I'm less than a noob.
Same connection details as the SSH connection.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
Everything that ends in .gz can be safely just moved off (deleting from the source) as they are archival and not being used.
rm /var/log/*.gz
Is that correct?Yes, that will remove them.
-
Filesystem Size Used Avail Use% Mounted on /dev/sda2 4.0G 3.1G 684M 83% / none 373M 72K 373M 1% /dev/shm /opt/xensource/packages/iso/XenCenter.iso 52M 52M 0 100% /var/xen/xc-install -
Much healthier.
-
We still need to see the logs
-
You might get some useful info from the XenCenter. In one of the earlier posts you had a Screencap of your XC. There were 5700+ notifications listed. I imagine if you click on that and goto Errors and Alerts buttons you will get some useful info. Alerts can be useful for finding the cause, Errors get a bit spammy in XC though they are useful as well.
Generally, the local storage not being connected is probably the first i would look at. That is bad. Then once that is working you need to figure out why your multipathd setup is borked. You said you changed some ip addresses; this is probably a major reason why it isnt working.
Have you tried logging into the switch this is all connected to and seeing if it is ok? -
@momurda said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
You might get some useful info from the XenCenter. In one of the earlier posts you had a Screencap of your XC. There were 5700+ notifications listed. I imagine if you click on that and goto Errors and Alerts buttons you will get some useful info. Alerts can be useful for finding the cause, Errors get a bit spammy in XC though they are useful as well.
Generally, the local storage not being connected is probably the first i would look at. That is bad. Then once that is working you need to figure out why your multipathd setup is borked. You said you changed some ip addresses; this is probably a major reason why it isnt working.
Have you tried logging into the switch this is all connected to and seeing if it is ok?Multipath is likely failing because the network is having issues.
-
What is the network setup on these servers and the SAN?
One switch, multiple switches? how many connections to each server? etc.
-
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
We still need to see the logs
[root@xen2 ~]# tail -n 100 messages tail: cannot open `messages' for reading: No such file or directory -
@momurda said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
You might get some useful info from the XenCenter. In one of the earlier posts you had a Screencap of your XC. There were 5700+ notifications listed. I imagine if you click on that and goto Errors and Alerts buttons you will get some useful info. Alerts can be useful for finding the cause, Errors get a bit spammy in XC though they are useful as well.
Generally, the local storage not being connected is probably the first i would look at. That is bad. Then once that is working you need to figure out why your multipathd setup is borked. You said you changed some ip addresses; this is probably a major reason why it isnt working.
Have you tried logging into the switch this is all connected to and seeing if it is ok?It's a few thousand of these.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
We still need to see the logs
[root@xen2 ~]# tail -n 100 messages tail: cannot open `messages' for reading: No such file or directorymight have to cd to the logs folder and try that tail again.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
It's a few thousand of these.
What about the 5 errors under Events?
-
@Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
What is the network setup on these servers and the SAN?
One switch, multiple switches? how many connections to each server? etc.
2 switches. I tried reaching out to the guy that set it up 6 years ago, and he said he doesn't even remember how he did it. It was the last time he did a XenServer and PowerVault setup. I can get you the networking info from the XC and from the PV, and maybe we can find info for the switch that way, but I have no documentation for it.
In my defense, I've been lobbying for a replacement for this setup. Hopefully, it's coming due now.
-
@Danp said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
It's a few thousand of these.
What about the 5 errors under Events?
All work I've been trying to do since it went down.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
We still need to see the logs
[root@xen2 ~]# tail -n 100 messages tail: cannot open `messages' for reading: No such file or directoryThat's not good. That means that something really went wrong. Probably the disk had filled up.
-
@Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
What is the network setup on these servers and the SAN?
One switch, multiple switches? how many connections to each server? etc.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
What is the network setup on these servers and the SAN?
One switch, multiple switches? how many connections to each server? etc.
2 switches. I tried reaching out to the guy that set it up 6 years ago, and he said he doesn't even remember how he did it. It was the last time he did a XenServer and PowerVault setup. I can get you the networking info from the XC and from the PV, and maybe we can find info for the switch that way, but I have no documentation for it.
In my defense, I've been lobbying for a replacement for this setup. Hopefully, it's coming due now.
XenServer is very good, but the version you have is old and not maintained. Having it with the PowerVault was never acceptable. That was a bad setup. XenServer is a very good choice. But if you don't have a XenServer team, you'll want something else. Of course, @scale is a good choice, but you already know that
Moving this to a Scale cluster would be a quick fix that would address both the compute and the storage needs and let you move to a far more reliable setup. -
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
What is the network setup on these servers and the SAN?
One switch, multiple switches? how many connections to each server? etc.
2 switches. I tried reaching out to the guy that set it up 6 years ago, and he said he doesn't even remember how he did it. It was the last time he did a XenServer and PowerVault setup. I can get you the networking info from the XC and from the PV, and maybe we can find info for the switch that way, but I have no documentation for it.
In my defense, I've been lobbying for a replacement for this setup. Hopefully, it's coming due now.
XenServer is very good, but the version you have is old and not maintained. Having it with the PowerVault was never acceptable. That was a bad setup. XenServer is a very good choice. But if you don't have a XenServer team, you'll want something else. Of course, @scale is a good choice, but you already know that
Moving this to a Scale cluster would be a quick fix that would address both the compute and the storage needs and let you move to a far more reliable setup.Is it possible to somehow spin up what we have on a Scale cluster as is?
