XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
It's a few thousand of these.
What about the 5 errors under Events?
-
@Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
What is the network setup on these servers and the SAN?
One switch, multiple switches? how many connections to each server? etc.
2 switches. I tried reaching out to the guy that set it up 6 years ago, and he said he doesn't even remember how he did it. It was the last time he did a XenServer and PowerVault setup. I can get you the networking info from the XC and from the PV, and maybe we can find info for the switch that way, but I have no documentation for it.
In my defense, I've been lobbying for a replacement for this setup. Hopefully, it's coming due now.
-
@Danp said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
It's a few thousand of these.
What about the 5 errors under Events?
All work I've been trying to do since it went down.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
We still need to see the logs
[root@xen2 ~]# tail -n 100 messages tail: cannot open `messages' for reading: No such file or directoryThat's not good. That means that something really went wrong. Probably the disk had filled up.
-
@Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
What is the network setup on these servers and the SAN?
One switch, multiple switches? how many connections to each server? etc.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
What is the network setup on these servers and the SAN?
One switch, multiple switches? how many connections to each server? etc.
2 switches. I tried reaching out to the guy that set it up 6 years ago, and he said he doesn't even remember how he did it. It was the last time he did a XenServer and PowerVault setup. I can get you the networking info from the XC and from the PV, and maybe we can find info for the switch that way, but I have no documentation for it.
In my defense, I've been lobbying for a replacement for this setup. Hopefully, it's coming due now.
XenServer is very good, but the version you have is old and not maintained. Having it with the PowerVault was never acceptable. That was a bad setup. XenServer is a very good choice. But if you don't have a XenServer team, you'll want something else. Of course, @scale is a good choice, but you already know that
Moving this to a Scale cluster would be a quick fix that would address both the compute and the storage needs and let you move to a far more reliable setup. -
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
What is the network setup on these servers and the SAN?
One switch, multiple switches? how many connections to each server? etc.
2 switches. I tried reaching out to the guy that set it up 6 years ago, and he said he doesn't even remember how he did it. It was the last time he did a XenServer and PowerVault setup. I can get you the networking info from the XC and from the PV, and maybe we can find info for the switch that way, but I have no documentation for it.
In my defense, I've been lobbying for a replacement for this setup. Hopefully, it's coming due now.
XenServer is very good, but the version you have is old and not maintained. Having it with the PowerVault was never acceptable. That was a bad setup. XenServer is a very good choice. But if you don't have a XenServer team, you'll want something else. Of course, @scale is a good choice, but you already know that
Moving this to a Scale cluster would be a quick fix that would address both the compute and the storage needs and let you move to a far more reliable setup.Is it possible to somehow spin up what we have on a Scale cluster as is?
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
What is the network setup on these servers and the SAN?
One switch, multiple switches? how many connections to each server? etc.
2 switches. I tried reaching out to the guy that set it up 6 years ago, and he said he doesn't even remember how he did it. It was the last time he did a XenServer and PowerVault setup. I can get you the networking info from the XC and from the PV, and maybe we can find info for the switch that way, but I have no documentation for it.
In my defense, I've been lobbying for a replacement for this setup. Hopefully, it's coming due now.
XenServer is very good, but the version you have is old and not maintained. Having it with the PowerVault was never acceptable. That was a bad setup. XenServer is a very good choice. But if you don't have a XenServer team, you'll want something else. Of course, @scale is a good choice, but you already know that
Moving this to a Scale cluster would be a quick fix that would address both the compute and the storage needs and let you move to a far more reliable setup.Is it possible to somehow spin up what we have on a Scale cluster as is?
Not quite, we need a means of grabbing the files off of it. So we either need a utility to do some conversion which I think Scale can help with; or we need to do a P2V process which requires us to get this working (even if just limping) in the mean time.
-
The iSCSI host ports on the SAN are 192.168.130.101-104. No gateway, no reference to a switch that I can find. I'm hoping a guy that's at lunch has some documentation to help out with that.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
The iSCSI host ports on the SAN are 192.168.130.101-104. No gateway, no reference to a switch that I can find. I'm hoping a guy that's at lunch has some documentation to help out with that.
I dont think that that is where we are stuck, so that's good info to have but not in our critical path at the moment (I don't think.)
-
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
We still need to see the logs
[root@xen2 ~]# tail -n 100 messages tail: cannot open `messages' for reading: No such file or directoryThat's not good. That means that something really went wrong. Probably the disk had filled up.
Is there another way to view log entries?
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
We still need to see the logs
[root@xen2 ~]# tail -n 100 messages tail: cannot open `messages' for reading: No such file or directoryThat's not good. That means that something really went wrong. Probably the disk had filled up.
Is there another way to view log entries?
Not if they are not there. That's not a log viewer, that's the logs itself. Are you use that you used the cd command to get into the right directory before trying to view the logs?
-
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
We still need to see the logs
[root@xen2 ~]# tail -n 100 messages tail: cannot open `messages' for reading: No such file or directoryThat's not good. That means that something really went wrong. Probably the disk had filled up.
Is there another way to view log entries?
Not if they are not there. That's not a log viewer, that's the logs itself. Are you use that you used the cd command to get into the right directory before trying to view the logs?
I used no cd command. I just tried "cd /var/logs" and was told there was no directory.
-
@scottalanmiller WAIT!!
-
Dec 27 13:04:50 xen2 multipathd: 360024e800070ed06000004314a967309: remaining ac tive paths: 1 Dec 27 13:04:50 xen2 xapi: [ info|xen2|1118516 UNIX /var/xapi/xapi|session.logou t D:8943f3fc0d25|xapi] Session.destroy trackid=08889875145ba350fd61700479bae5a1 Dec 27 13:04:50 xen2 xapi: [ info|xen2|1118517 UNIX /var/xapi/xapi|session.login _with_password D:fc3ccd82666a|xapi] Session.create trackid=f2aea39683d7b6a01a324 e173f3dd8b6 pool=false uname=root is_local_superuser=true auth_user_sid= parent= trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:04:51 xen2 kernel: [17525.713536] sd 6:0:0:1: [sdi] Unhandled sense co de Dec 27 13:04:51 xen2 kernel: [17525.713539] sd 6:0:0:1: [sdi] Result: hostbyte=D ID_OK driverbyte=DRIVER_SENSE Dec 27 13:04:51 xen2 kernel: [17525.713541] sd 6:0:0:1: [sdi] Sense Key : Hardwa re Error [current] Dec 27 13:04:51 xen2 kernel: [17525.713544] sd 6:0:0:1: [sdi] <<vendor>> ASC=0x8 4 ASCQ=0x0ASC=0x84 ASCQ=0x0 Dec 27 13:04:51 xen2 kernel: [17525.713548] sd 6:0:0:1: [sdi] CDB: Read(10): 28 00 ae ff ff 80 00 00 08 00 Dec 27 13:04:51 xen2 kernel: [17525.713553] end_request: I/O error, dev sdi, sec tor 2936012672 Dec 27 13:04:51 xen2 kernel: [17525.713569] device-mapper: multipath: Failing pa th 8:128. Dec 27 13:04:51 xen2 xapi: [ info|xen2|1118555 INET 0.0.0.0:80|session.logout D: c734fb0019a4|xapi] Session.destroy trackid=b33614b35799e4591630faf1215fb656 Dec 27 13:04:51 xen2 multipathd: 8:128: mark as failed Dec 27 13:04:51 xen2 multipathd: 360024e800070ed06000004314a967309: Entering rec overy mode: max_retries=15 Dec 27 13:04:51 xen2 multipathd: 360024e800070ed06000004314a967309: remaining ac tive paths: 0 Dec 27 13:04:51 xen2 multipathd: Path event for 360024e800070ed06000004314a96730 9, request call of mpathcount Dec 27 13:04:51 xen2 xapi: [ info|xen2|1118565 UNIX /var/xapi/xapi|session.logou t D:5e8426752b6a|xapi] Session.destroy trackid=f2aea39683d7b6a01a324e173f3dd8b6 Dec 27 13:04:51 xen2 xapi: [ info|xen2|1118566 UNIX /var/xapi/xapi|session.login _with_password D:7bf7776c558a|xapi] Session.create trackid=0b8b2266594c81ed0f491 512ed56da5b pool=false uname=root is_local_superuser=true auth_user_sid= parent= trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:04:51 xen2 multipathd: 360024e800070ed06000004314a967309: Entering rec overy mode: max_retries=15 Dec 27 13:04:51 xen2 xapi: [ info|xen2|1118609 UNIX /var/xapi/xapi|session.logou t D:b0f10b9ab800|xapi] Session.destroy trackid=0b8b2266594c81ed0f491512ed56da5b Dec 27 13:04:51 xen2 xapi: [ info|xen2|1118610 UNIX /var/xapi/xapi|session.login _with_password D:83a1d8c13406|xapi] Session.create trackid=57c35017dc0b58cba5006 fab885d6c42 pool=false uname=root is_local_superuser=true auth_user_sid= parent= trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:04:52 xen2 xapi: [ info|xen2|1118649 INET 0.0.0.0:80|session.logout D: 80db80fdafff|xapi] Session.destroy trackid=a6e94a7ff69b574b678611e81606d239 Dec 27 13:04:52 xen2 xapi: [ info|xen2|1118654 UNIX /var/xapi/xapi|session.logou t D:29c302273758|xapi] Session.destroy trackid=57c35017dc0b58cba5006fab885d6c42 Dec 27 13:04:52 xen2 multipathd: 360024e800070ed06000004314a967309: sdl - rdac c hecker reports path is ghost Dec 27 13:04:52 xen2 multipathd: Path event for 360024e800070ed06000004314a96730 9, request call of mpathcount Dec 27 13:04:52 xen2 multipathd: 8:176: reinstated Dec 27 13:04:52 xen2 multipathd: 360024e800070ed06000004314a967309: queue_if_no_ path enabled Dec 27 13:04:52 xen2 multipathd: 360024e800070ed06000004314a967309: Recovered to normal mode Dec 27 13:04:52 xen2 multipathd: 360024e800070ed06000004314a967309: remaining ac tive paths: 1 Dec 27 13:04:52 xen2 xapi: [ info|xen2|1118668 UNIX /var/xapi/xapi|session.login _with_password D:8afb60727358|xapi] Session.create trackid=492e0a0a5edb18140a204 dcf801b0eb0 pool=false uname=root is_local_superuser=true auth_user_sid= parent= trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:04:52 xen2 multipathd: 360024e800070ed06000004314a967309: switch to pa th group #4 Dec 27 13:04:52 xen2 kernel: [17527.512748] sd 7:0:0:1: rdac: array MDS-Spindle0 1, ctlr 1, queueing MODE_SELECT command Dec 27 13:04:53 xen2 xapi: [ info|xen2|1118702 UNIX /var/xapi/xapi|session.logou t D:d0126e6b3bad|xapi] Session.destroy trackid=492e0a0a5edb18140a204dcf801b0eb0 Dec 27 13:04:53 xen2 kernel: [17528.017140] sd 7:0:0:1: rdac: array MDS-Spindle0 1, ctlr 1, MODE_SELECT returned with sense 05/91/36 Dec 27 13:04:53 xen2 kernel: [17528.017153] sd 7:0:0:1: rdac: array MDS-Spindle0 1, ctlr 1, retrying MODE_SELECT command Dec 27 13:04:53 xen2 kernel: [17528.303402] sd 7:0:0:1: rdac: array MDS-Spindle0 1, ctlr 1, MODE_SELECT completed Dec 27 13:04:53 xen2 multipathd: 360024e800070ed06000004314a967309: sdf - rdac c hecker reports path is up Dec 27 13:04:53 xen2 multipathd: Path event for 360024e800070ed06000004314a96730 9, request call of mpathcount Dec 27 13:04:53 xen2 multipathd: 8:80: reinstated Dec 27 13:04:53 xen2 multipathd: 360024e800070ed06000004314a967309: remaining ac tive paths: 2 Dec 27 13:04:54 xen2 xapi: [ info|xen2|1118703 UNIX /var/xapi/xapi|session.login _with_password D:cd022f51a23b|xapi] Session.create trackid=78096f7ecdee23b2dae33 38911791e6f pool=false uname=root is_local_superuser=true auth_user_sid= parent= trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:04:54 xen2 multipathd: 360024e800070ed06000004314a967309: load table [ 0 2936012800 multipath 3 pg_init_retries 50 queue_if_no_path 1 rdac 3 3 round-ro bin 0 1 1 8:128 1000 round-robin 0 1 1 8:32 1000 round-robin 0 2 1 8:80 1000 8:1 76 1000] Dec 27 13:04:54 xen2 multipathd: Path event for 360024e800070ed06000004314a96730 9, request call of mpathcount Dec 27 13:04:54 xen2 multipathd: Path event for 360024e800070ed06000004314a96730 9, request call of mpathcount Dec 27 13:04:54 xen2 kernel: [17528.816240] sd 7:0:0:1: [sdl] Unhandled sense co de Dec 27 13:04:54 xen2 kernel: [17528.816243] sd 7:0:0:1: [sdl] Result: hostbyte=D ID_OK driverbyte=DRIVER_SENSE Dec 27 13:04:54 xen2 kernel: [17528.816245] sd 7:0:0:1: [sdl] Sense Key : Hardwa re Error [current] Dec 27 13:04:54 xen2 kernel: [17528.816248] sd 7:0:0:1: [sdl] <<vendor>> ASC=0x8 4 ASCQ=0x0ASC=0x84 ASCQ=0x0 Dec 27 13:04:54 xen2 kernel: [17528.816252] sd 7:0:0:1: [sdl] CDB: Read(10): 28 00 ae ff ff 80 00 00 08 00 Dec 27 13:04:54 xen2 kernel: [17528.816257] end_request: I/O error, dev sdl, sec tor 2936012672 Dec 27 13:04:54 xen2 kernel: [17528.816274] device-mapper: multipath: Failing pa th 8:176. Dec 27 13:04:54 xen2 kernel: [17528.817236] device-mapper: multipath: Failing pa th 8:128. Dec 27 13:04:54 xen2 kernel: [17528.817315] device-mapper: multipath: Failing pa th 8:32. Dec 27 13:04:54 xen2 xapi: [ info|xen2|1118753 UNIX /var/xapi/xapi|session.logou t D:5eabc2a6317b|xapi] Session.destroy trackid=78096f7ecdee23b2dae3338911791e6f Dec 27 13:04:54 xen2 xapi: [ info|xen2|1118755 UNIX /var/xapi/xapi|session.login _with_password D:1894b4b17ab8|xapi] Session.create trackid=920f0d86a645a2f1f126e d2b0fc12f87 pool=false uname=root is_local_superuser=true auth_user_sid= parent= trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:04:54 xen2 xapi: [ info|xen2|1118776 INET 0.0.0.0:80|session.logout D: 3c2cf4cc964b|xapi] Session.destroy trackid=e7b050fd41937a5158bf0b2ca39b1659 Dec 27 13:04:54 xen2 kernel: [17529.332325] sd 5:0:0:1: [sdf] Unhandled sense co de Dec 27 13:04:54 xen2 kernel: [17529.332329] sd 5:0:0:1: [sdf] Result: hostbyte=D ID_OK driverbyte=DRIVER_SENSE Dec 27 13:04:54 xen2 kernel: [17529.332331] sd 5:0:0:1: [sdf] Sense Key : Hardwa re Error [current] Dec 27 13:04:54 xen2 kernel: [17529.332334] sd 5:0:0:1: [sdf] <<vendor>> ASC=0x8 4 ASCQ=0x0ASC=0x84 ASCQ=0x0 Dec 27 13:04:54 xen2 kernel: [17529.332337] sd 5:0:0:1: [sdf] CDB: Read(10): 28 00 ae ff ff 80 00 00 08 00 Dec 27 13:04:54 xen2 kernel: [17529.332343] end_request: I/O error, dev sdf, sec tor 2936012672 Dec 27 13:04:54 xen2 kernel: [17529.332359] device-mapper: multipath: Failing pa th 8:80. Dec 27 13:04:54 xen2 multipathd: 8:80: mark as failed Dec 27 13:04:54 xen2 multipathd: 360024e800070ed06000004314a967309: remaining ac tive paths: 1 Dec 27 13:04:54 xen2 multipathd: Path event for 360024e800070ed06000004314a96730 9, request call of mpathcount Dec 27 13:04:55 xen2 xapi: [ info|xen2|1118805 UNIX /var/xapi/xapi|session.logou t D:41ef5f113fba|xapi] Session.destroy trackid=920f0d86a645a2f1f126ed2b0fc12f87 Dec 27 13:04:55 xen2 xapi: [ info|xen2|1118806 UNIX /var/xapi/xapi|session.login _with_password D:8b9980e8c1ea|xapi] Session.create trackid=8de1347f37c83b6954ea0 9681335db4e pool=false uname=root is_local_superuser=true auth_user_sid= parent= trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:04:55 xen2 multipathd: 360024e800070ed06000004314a967309: sdc - rdac c hecker reports path is ghost Dec 27 13:04:55 xen2 multipathd: Path event for 360024e800070ed06000004314a96730 9, request call of mpathcount Dec 27 13:04:55 xen2 multipathd: 8:32: reinstated Dec 27 13:04:55 xen2 multipathd: 360024e800070ed06000004314a967309: remaining ac tive paths: 2 Dec 27 13:04:55 xen2 multipathd: 360024e800070ed06000004314a967309: switch to pa th group #2 Dec 27 13:04:55 xen2 multipathd: 8:176: mark as failed Dec 27 13:04:55 xen2 multipathd: 360024e800070ed06000004314a967309: remaining ac tive paths: 1 Dec 27 13:04:55 xen2 multipathd: Path event for 360024e800070ed06000004314a96730 9, request call of mpathcount Dec 27 13:04:55 xen2 kernel: [17529.844849] sd 7:0:0:1: [sdl] Unhandled sense co de Dec 27 13:04:55 xen2 kernel: [17529.844852] sd 7:0:0:1: [sdl] Result: hostbyte=D ID_OK driverbyte=DRIVER_SENSE Dec 27 13:04:55 xen2 kernel: [17529.844855] sd 7:0:0:1: [sdl] Sense Key : Hardwa re Error [current] Dec 27 13:04:55 xen2 kernel: [17529.844857] sd 7:0:0:1: [sdl] <<vendor>> ASC=0x8 4 ASCQ=0x0ASC=0x84 ASCQ=0x0 Dec 27 13:04:55 xen2 kernel: [17529.844861] sd 7:0:0:1: [sdl] CDB: Read(10): 28 00 ae ff ff 80 00 00 08 00 Dec 27 13:04:55 xen2 kernel: [17529.844866] end_request: I/O error, dev sdl, sec tor 2936012672 Dec 27 13:04:55 xen2 kernel: [17529.844879] device-mapper: multipath: Failing pa th 8:176. Dec 27 13:04:55 xen2 kernel: [17530.081711] sd 4:0:0:1: rdac: array MDS-Spindle0 1, ctlr 0, queueing MODE_SELECT command Dec 27 13:04:55 xen2 xapi: [ info|xen2|1118849 UNIX /var/xapi/xapi|session.logou t D:0494e32893e6|xapi] Session.destroy trackid=8de1347f37c83b6954ea09681335db4e Dec 27 13:04:55 xen2 xapi: [ info|xen2|1118850 UNIX /var/xapi/xapi|session.login _with_password D:6baeffeb13e0|xapi] Session.create trackid=b043dd9f380b982f1e8aa 32967a8de2b pool=false uname=root is_local_superuser=true auth_user_sid= parent= trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:04:55 xen2 xapi: [ info|xen2|1118852 INET 0.0.0.0:80|session.logout D: 97bc80b93f75|xapi] Session.destroy trackid=53e2afc20c1a0ab616b48da0c7004030 Dec 27 13:04:55 xen2 kernel: [17530.329388] sd 4:0:0:1: rdac: array MDS-Spindle0 1, ctlr 0, MODE_SELECT completed Dec 27 13:04:55 xen2 xapi: [ info|xen2|1118884 UNIX /var/xapi/xapi|session.logou t D:15d77c89ee0a|xapi] Session.destroy trackid=b043dd9f380b982f1e8aa32967a8de2b Dec 27 13:04:56 xen2 kernel: [17530.841619] sd 4:0:0:1: [sdc] Unhandled sense co de Dec 27 13:04:56 xen2 kernel: [17530.841623] sd 4:0:0:1: [sdc] Result: hostbyte=D ID_OK driverbyte=DRIVER_SENSE Dec 27 13:04:56 xen2 kernel: [17530.841628] sd 4:0:0:1: [sdc] Sense Key : Hardwa re Error [current] Dec 27 13:04:56 xen2 kernel: [17530.841632] sd 4:0:0:1: [sdc] <<vendor>> ASC=0x8 4 ASCQ=0x0ASC=0x84 ASCQ=0x0 Dec 27 13:04:56 xen2 kernel: [17530.841638] sd 4:0:0:1: [sdc] CDB: Read(10): 28 00 ae ff ff 80 00 00 08 00 Dec 27 13:04:56 xen2 kernel: [17530.841648] end_request: I/O error, dev sdc, sec tor 2936012672 Dec 27 13:04:56 xen2 kernel: [17530.841672] device-mapper: multipath: Failing pa th 8:32. Dec 27 13:04:56 xen2 multipathd: 360024e800070ed06000004314a967309: sdi - rdac c hecker reports path is up Dec 27 13:04:56 xen2 multipathd: Path event for 360024e800070ed06000004314a96730 9, request call of mpathcount Dec 27 13:04:56 xen2 multipathd: 8:128: reinstated Dec 27 13:04:56 xen2 multipathd: 360024e800070ed06000004314a967309: remaining ac tive paths: 2 Dec 27 13:04:56 xen2 xapi: [ info|xen2|1118889 UNIX /var/xapi/xapi|session.login _with_password D:f3f2ba7c9b76|xapi] Session.create trackid=1f938a40d5b26b9761de4 6230d454ee5 pool=false uname=root is_local_superuser=true auth_user_sid= parent= trackid=9834f5af41c964e225f24279aefe4e49 -
There you go..
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
We still need to see the logs
[root@xen2 ~]# tail -n 100 messages tail: cannot open `messages' for reading: No such file or directoryThat's not good. That means that something really went wrong. Probably the disk had filled up.
Is there another way to view log entries?
Not if they are not there. That's not a log viewer, that's the logs itself. Are you use that you used the cd command to get into the right directory before trying to view the logs?
I used no cd command. I just tried "cd /var/logs" and was told there was no directory.
We are definitely seeing a problem here
-
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
We still need to see the logs
[root@xen2 ~]# tail -n 100 messages tail: cannot open `messages' for reading: No such file or directoryThat's not good. That means that something really went wrong. Probably the disk had filled up.
Is there another way to view log entries?
Not if they are not there. That's not a log viewer, that's the logs itself. Are you use that you used the cd command to get into the right directory before trying to view the logs?
I used no cd command. I just tried "cd /var/logs" and was told there was no directory.
We are definitely seeing a problem here
Scott, why is the formatting so messed up?
-
@Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
We still need to see the logs
[root@xen2 ~]# tail -n 100 messages tail: cannot open `messages' for reading: No such file or directoryThat's not good. That means that something really went wrong. Probably the disk had filled up.
Is there another way to view log entries?
Not if they are not there. That's not a log viewer, that's the logs itself. Are you use that you used the cd command to get into the right directory before trying to view the logs?
I used no cd command. I just tried "cd /var/logs" and was told there was no directory.
We are definitely seeing a problem here
Scott, why is the formatting so messed up?
No idea, but it's not from ML, that's from his cut and paste. Those spaces are all in his original.
-
Dec 27 13:13:16 xen2 last message repeated 2 times Dec 27 13:13:16 xen2 kernel: [18030.894547] device-mapper: multipath: Failing path 8:128. Dec 27 13:13:16 xen2 kernel: [18030.894934] device-mapper: multipath: Failing path 8:80. Dec 27 13:13:16 xen2 kernel: [18030.895002] device-mapper: multipath: Failing path 8:176. Dec 27 13:13:16 xen2 xapi: [ info|xen2|1151320 UNIX /var/xapi/xapi|session.logout D:9b0536d5f01c|xapi] Session.destroy trackid=e8dec3c23c8d667dfcfbfd772ecabd9e Dec 27 13:13:16 xen2 xapi: [ info|xen2|1151321 UNIX /var/xapi/xapi|session.login_with_password D:72d677979b15|xapi] Session.create trackid=2db03e1031b1b474ab4e3a61a73c4824 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:13:16 xen2 kernel: [18031.409957] sd 4:0:0:1: [sdc] Unhandled sense code Dec 27 13:13:16 xen2 kernel: [18031.409960] sd 4:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Dec 27 13:13:16 xen2 kernel: [18031.409963] sd 4:0:0:1: [sdc] Sense Key : Hardware Error [current] Dec 27 13:13:16 xen2 kernel: [18031.409965] sd 4:0:0:1: [sdc] <<vendor>> ASC=0x84 ASCQ=0x0ASC=0x84 ASCQ=0x0 Dec 27 13:13:16 xen2 kernel: [18031.409969] sd 4:0:0:1: [sdc] CDB: Read(10): 28 00 ae ff ff 80 00 00 08 00 Dec 27 13:13:16 xen2 kernel: [18031.409974] end_request: I/O error, dev sdc, sector 2936012672 Dec 27 13:13:16 xen2 kernel: [18031.409989] device-mapper: multipath: Failing path 8:32. Dec 27 13:13:16 xen2 xapi: [ info|xen2|1151359 INET 0.0.0.0:80|session.logout D:6b5f0e1950d3|xapi] Session.destroy trackid=95373089b3a53ad07d348ea483840e40 Dec 27 13:13:16 xen2 multipathd: 8:32: mark as failed Dec 27 13:13:16 xen2 multipathd: 360024e800070ed06000004314a967309: Entering recovery mode: max_retries=15 Dec 27 13:13:16 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 0 Dec 27 13:13:16 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount Dec 27 13:13:16 xen2 xapi: [ info|xen2|1151365 UNIX /var/xapi/xapi|session.logout D:7808df6ecf1f|xapi] Session.destroy trackid=2db03e1031b1b474ab4e3a61a73c4824 Dec 27 13:13:16 xen2 xapi: [ info|xen2|1151366 UNIX /var/xapi/xapi|session.login_with_password D:107e4867f954|xapi] Session.create trackid=398819a8505258b63db68cd0d33af4b1 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:13:17 xen2 multipathd: 360024e800070ed06000004314a967309: sdi - rdac checker reports path is up Dec 27 13:13:17 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount Dec 27 13:13:17 xen2 multipathd: 8:128: reinstated Dec 27 13:13:17 xen2 multipathd: 360024e800070ed06000004314a967309: queue_if_no_path enabled Dec 27 13:13:17 xen2 multipathd: 360024e800070ed06000004314a967309: Recovered to normal mode Dec 27 13:13:17 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 1 Dec 27 13:13:17 xen2 xapi: [ info|xen2|1151408 UNIX /var/xapi/xapi|session.logout D:96b0ad24c6c1|xapi] Session.destroy trackid=398819a8505258b63db68cd0d33af4b1 Dec 27 13:13:17 xen2 xapi: [ info|xen2|1151409 UNIX /var/xapi/xapi|session.login_with_password D:f87fb7e4fce8|xapi] Session.create trackid=a797b1b90698608ddd5f6919804b5484 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:13:17 xen2 kernel: [18032.440371] sd 6:0:0:1: [sdi] Unhandled sense code Dec 27 13:13:17 xen2 kernel: [18032.440374] sd 6:0:0:1: [sdi] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Dec 27 13:13:17 xen2 kernel: [18032.440376] sd 6:0:0:1: [sdi] Sense Key : Hardware Error [current] Dec 27 13:13:17 xen2 kernel: [18032.440379] sd 6:0:0:1: [sdi] <<vendor>> ASC=0x84 ASCQ=0x0ASC=0x84 ASCQ=0x0 Dec 27 13:13:17 xen2 kernel: [18032.440382] sd 6:0:0:1: [sdi] CDB: Read(10): 28 00 ae ff ff 80 00 00 08 00 Dec 27 13:13:17 xen2 kernel: [18032.440387] end_request: I/O error, dev sdi, sector 2936012672 Dec 27 13:13:17 xen2 kernel: [18032.440404] device-mapper: multipath: Failing path 8:128. Dec 27 13:13:17 xen2 multipathd: 8:128: mark as failed Dec 27 13:13:17 xen2 multipathd: 360024e800070ed06000004314a967309: Entering recovery mode: max_retries=15 Dec 27 13:13:17 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 0 Dec 27 13:13:17 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount Dec 27 13:13:17 xen2 xapi: [ info|xen2|1151447 UNIX /var/xapi/xapi|session.logout D:acca552975a2|xapi] Session.destroy trackid=a797b1b90698608ddd5f6919804b5484 Dec 27 13:13:17 xen2 xapi: [ info|xen2|1151448 UNIX /var/xapi/xapi|session.login_with_password D:5f3e3fccf3d5|xapi] Session.create trackid=53cb15330f2848d21867cbf051db9ed9 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:13:18 xen2 multipathd: 360024e800070ed06000004314a967309: sdl - rdac checker reports path is ghost Dec 27 13:13:18 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount Dec 27 13:13:18 xen2 multipathd: 8:176: reinstated Dec 27 13:13:18 xen2 multipathd: 360024e800070ed06000004314a967309: queue_if_no_path enabled Dec 27 13:13:18 xen2 multipathd: 360024e800070ed06000004314a967309: Recovered to normal mode Dec 27 13:13:18 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 1 Dec 27 13:13:18 xen2 multipathd: 360024e800070ed06000004314a967309: switch to path group #4 Dec 27 13:13:18 xen2 kernel: [18033.217827] sd 7:0:0:1: rdac: array MDS-Spindle01, ctlr 1, queueing MODE_SELECT command Dec 27 13:13:18 xen2 xapi: [ info|xen2|1151482 UNIX /var/xapi/xapi|session.logout D:39dea3d23bb2|xapi] Session.destroy trackid=53cb15330f2848d21867cbf051db9ed9 Dec 27 13:13:18 xen2 xapi: [ info|xen2|1151483 UNIX /var/xapi/xapi|session.login_with_password D:487d0c440a6f|xapi] Session.create trackid=31c6550a7ad5c485e04e1ce80f3588dd pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:13:18 xen2 kernel: [18033.500501] sd 7:0:0:1: rdac: array MDS-Spindle01, ctlr 1, MODE_SELECT completed Dec 27 13:13:19 xen2 xapi: [ info|xen2|1151517 UNIX /var/xapi/xapi|session.logout D:6b32233bc36f|xapi] Session.destroy trackid=31c6550a7ad5c485e04e1ce80f3588dd Dec 27 13:13:19 xen2 kernel: [18034.028373] sd 7:0:0:1: [sdl] Unhandled sense code Dec 27 13:13:19 xen2 kernel: [18034.028376] sd 7:0:0:1: [sdl] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Dec 27 13:13:19 xen2 kernel: [18034.028379] sd 7:0:0:1: [sdl] Sense Key : Hardware Error [current] Dec 27 13:13:19 xen2 kernel: [18034.028381] sd 7:0:0:1: [sdl] <<vendor>> ASC=0x84 ASCQ=0x0ASC=0x84 ASCQ=0x0 Dec 27 13:13:19 xen2 kernel: [18034.028385] sd 7:0:0:1: [sdl] CDB: Read(10): 28 00 ae ff ff 80 00 00 08 00 Dec 27 13:13:19 xen2 kernel: [18034.028390] end_request: I/O error, dev sdl, sector 2936012672 Dec 27 13:13:19 xen2 kernel: [18034.028406] device-mapper: multipath: Failing path 8:176. Dec 27 13:13:19 xen2 xapi: [ info|xen2|1151537 INET 0.0.0.0:80|session.logout D:7e74bbbad814|xapi] Session.destroy trackid=57a035032e821d9289ab2ddc9c56ef23 Dec 27 13:13:19 xen2 multipathd: 8:176: mark as failed Dec 27 13:13:19 xen2 multipathd: 360024e800070ed06000004314a967309: Entering recovery mode: max_retries=15 Dec 27 13:13:19 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 0 Dec 27 13:13:19 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount Dec 27 13:13:19 xen2 xapi: [ info|xen2|1151540 UNIX /var/xapi/xapi|session.login_with_password D:d750b0705d7f|xapi] Session.create trackid=86b83edaef195351a3b0fab7cc6cc21e pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:13:20 xen2 xapi: [ info|xen2|1151585 UNIX /var/xapi/xapi|session.logout D:3e57cedfef7a|xapi] Session.destroy trackid=86b83edaef195351a3b0fab7cc6cc21e Dec 27 13:13:20 xen2 multipathd: 360024e800070ed06000004314a967309: sdf - rdac checker reports path is up Dec 27 13:13:20 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount Dec 27 13:13:20 xen2 multipathd: 8:80: reinstated Dec 27 13:13:20 xen2 multipathd: 360024e800070ed06000004314a967309: queue_if_no_path enabled Dec 27 13:13:20 xen2 multipathd: 360024e800070ed06000004314a967309: Recovered to normal mode Dec 27 13:13:20 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 1 Dec 27 13:13:20 xen2 multipathd: 360024e800070ed06000004314a967309: switch to path group #3 Dec 27 13:13:20 xen2 xapi: [ info|xen2|1151605 UNIX /var/xapi/xapi|session.login_with_password D:1379b8d2955b|xapi] Session.create trackid=3fa7db291a48d0f98ea6e364b1dccd5b pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:13:20 xen2 xapi: [ info|xen2|1151611 INET 0.0.0.0:80|session.logout D:e6afd1da2cf0|xapi] Session.destroy trackid=a246f22117a6ddf72bd492a7e8505ddd Dec 27 13:13:21 xen2 kernel: [18035.743234] sd 5:0:0:1: [sdf] Unhandled sense code Dec 27 13:13:21 xen2 kernel: [18035.743237] sd 5:0:0:1: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Dec 27 13:13:21 xen2 kernel: [18035.743240] sd 5:0:0:1: [sdf] Sense Key : Hardware Error [current] Dec 27 13:13:21 xen2 kernel: [18035.743242] sd 5:0:0:1: [sdf] <<vendor>> ASC=0x84 ASCQ=0x0ASC=0x84 ASCQ=0x0 Dec 27 13:13:21 xen2 kernel: [18035.743246] sd 5:0:0:1: [sdf] CDB: Read(10): 28 00 ae ff ff 80 00 00 08 00 Dec 27 13:13:21 xen2 kernel: [18035.743251] end_request: I/O error, dev sdf, sector 2936012672 Dec 27 13:13:21 xen2 kernel: [18035.743266] device-mapper: multipath: Failing path 8:80. Dec 27 13:13:21 xen2 multipathd: 8:80: mark as failed Dec 27 13:13:21 xen2 multipathd: 360024e800070ed06000004314a967309: Entering recovery mode: max_retries=15 Dec 27 13:13:21 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 0 Dec 27 13:13:21 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount Dec 27 13:13:21 xen2 xapi: [ info|xen2|1151653 UNIX /var/xapi/xapi|session.logout D:57acea842a91|xapi] Session.destroy trackid=3fa7db291a48d0f98ea6e364b1dccd5b Dec 27 13:13:21 xen2 xapi: [ info|xen2|1151654 UNIX /var/xapi/xapi|session.login_with_password D:e547785afb43|xapi] Session.create trackid=8fd799a70bdc34fb2a4d8bd676873ede pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:13:21 xen2 multipathd: 360024e800070ed06000004314a967309: sdc - rdac checker reports path is ghost Dec 27 13:13:21 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount Dec 27 13:13:21 xen2 multipathd: 8:32: reinstated Dec 27 13:13:21 xen2 multipathd: 360024e800070ed06000004314a967309: queue_if_no_path enabled Dec 27 13:13:21 xen2 multipathd: 360024e800070ed06000004314a967309: Recovered to normal mode Dec 27 13:13:21 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 1 Dec 27 13:13:21 xen2 multipathd: 360024e800070ed06000004314a967309: switch to path group #2 Dec 27 13:13:21 xen2 kernel: [18036.256313] sd 4:0:0:1: rdac: array MDS-Spindle01, ctlr 0, queueing MODE_SELECT command Dec 27 13:13:21 xen2 kernel: [18036.525838] sd 4:0:0:1: rdac: array MDS-Spindle01, ctlr 0, MODE_SELECT completed Dec 27 13:13:22 xen2 xapi: [ info|xen2|1151678 UNIX /var/xapi/xapi|session.logout D:34466f735dad|xapi] Session.destroy trackid=8fd799a70bdc34fb2a4d8bd676873ede Dec 27 13:13:22 xen2 xapi: [ info|xen2|1151679 UNIX /var/xapi/xapi|session.login_with_password D:cd7b037c4106|xapi] Session.create trackid=f258157e7fefe65077464bbd3c912521 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
