XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!
-
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
We still need to see the logs
[root@xen2 ~]# tail -n 100 messages tail: cannot open `messages' for reading: No such file or directoryThat's not good. That means that something really went wrong. Probably the disk had filled up.
Is there another way to view log entries?
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
We still need to see the logs
[root@xen2 ~]# tail -n 100 messages tail: cannot open `messages' for reading: No such file or directoryThat's not good. That means that something really went wrong. Probably the disk had filled up.
Is there another way to view log entries?
Not if they are not there. That's not a log viewer, that's the logs itself. Are you use that you used the cd command to get into the right directory before trying to view the logs?
-
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
We still need to see the logs
[root@xen2 ~]# tail -n 100 messages tail: cannot open `messages' for reading: No such file or directoryThat's not good. That means that something really went wrong. Probably the disk had filled up.
Is there another way to view log entries?
Not if they are not there. That's not a log viewer, that's the logs itself. Are you use that you used the cd command to get into the right directory before trying to view the logs?
I used no cd command. I just tried "cd /var/logs" and was told there was no directory.
-
@scottalanmiller WAIT!!
-
Dec 27 13:04:50 xen2 multipathd: 360024e800070ed06000004314a967309: remaining ac tive paths: 1 Dec 27 13:04:50 xen2 xapi: [ info|xen2|1118516 UNIX /var/xapi/xapi|session.logou t D:8943f3fc0d25|xapi] Session.destroy trackid=08889875145ba350fd61700479bae5a1 Dec 27 13:04:50 xen2 xapi: [ info|xen2|1118517 UNIX /var/xapi/xapi|session.login _with_password D:fc3ccd82666a|xapi] Session.create trackid=f2aea39683d7b6a01a324 e173f3dd8b6 pool=false uname=root is_local_superuser=true auth_user_sid= parent= trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:04:51 xen2 kernel: [17525.713536] sd 6:0:0:1: [sdi] Unhandled sense co de Dec 27 13:04:51 xen2 kernel: [17525.713539] sd 6:0:0:1: [sdi] Result: hostbyte=D ID_OK driverbyte=DRIVER_SENSE Dec 27 13:04:51 xen2 kernel: [17525.713541] sd 6:0:0:1: [sdi] Sense Key : Hardwa re Error [current] Dec 27 13:04:51 xen2 kernel: [17525.713544] sd 6:0:0:1: [sdi] <<vendor>> ASC=0x8 4 ASCQ=0x0ASC=0x84 ASCQ=0x0 Dec 27 13:04:51 xen2 kernel: [17525.713548] sd 6:0:0:1: [sdi] CDB: Read(10): 28 00 ae ff ff 80 00 00 08 00 Dec 27 13:04:51 xen2 kernel: [17525.713553] end_request: I/O error, dev sdi, sec tor 2936012672 Dec 27 13:04:51 xen2 kernel: [17525.713569] device-mapper: multipath: Failing pa th 8:128. Dec 27 13:04:51 xen2 xapi: [ info|xen2|1118555 INET 0.0.0.0:80|session.logout D: c734fb0019a4|xapi] Session.destroy trackid=b33614b35799e4591630faf1215fb656 Dec 27 13:04:51 xen2 multipathd: 8:128: mark as failed Dec 27 13:04:51 xen2 multipathd: 360024e800070ed06000004314a967309: Entering rec overy mode: max_retries=15 Dec 27 13:04:51 xen2 multipathd: 360024e800070ed06000004314a967309: remaining ac tive paths: 0 Dec 27 13:04:51 xen2 multipathd: Path event for 360024e800070ed06000004314a96730 9, request call of mpathcount Dec 27 13:04:51 xen2 xapi: [ info|xen2|1118565 UNIX /var/xapi/xapi|session.logou t D:5e8426752b6a|xapi] Session.destroy trackid=f2aea39683d7b6a01a324e173f3dd8b6 Dec 27 13:04:51 xen2 xapi: [ info|xen2|1118566 UNIX /var/xapi/xapi|session.login _with_password D:7bf7776c558a|xapi] Session.create trackid=0b8b2266594c81ed0f491 512ed56da5b pool=false uname=root is_local_superuser=true auth_user_sid= parent= trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:04:51 xen2 multipathd: 360024e800070ed06000004314a967309: Entering rec overy mode: max_retries=15 Dec 27 13:04:51 xen2 xapi: [ info|xen2|1118609 UNIX /var/xapi/xapi|session.logou t D:b0f10b9ab800|xapi] Session.destroy trackid=0b8b2266594c81ed0f491512ed56da5b Dec 27 13:04:51 xen2 xapi: [ info|xen2|1118610 UNIX /var/xapi/xapi|session.login _with_password D:83a1d8c13406|xapi] Session.create trackid=57c35017dc0b58cba5006 fab885d6c42 pool=false uname=root is_local_superuser=true auth_user_sid= parent= trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:04:52 xen2 xapi: [ info|xen2|1118649 INET 0.0.0.0:80|session.logout D: 80db80fdafff|xapi] Session.destroy trackid=a6e94a7ff69b574b678611e81606d239 Dec 27 13:04:52 xen2 xapi: [ info|xen2|1118654 UNIX /var/xapi/xapi|session.logou t D:29c302273758|xapi] Session.destroy trackid=57c35017dc0b58cba5006fab885d6c42 Dec 27 13:04:52 xen2 multipathd: 360024e800070ed06000004314a967309: sdl - rdac c hecker reports path is ghost Dec 27 13:04:52 xen2 multipathd: Path event for 360024e800070ed06000004314a96730 9, request call of mpathcount Dec 27 13:04:52 xen2 multipathd: 8:176: reinstated Dec 27 13:04:52 xen2 multipathd: 360024e800070ed06000004314a967309: queue_if_no_ path enabled Dec 27 13:04:52 xen2 multipathd: 360024e800070ed06000004314a967309: Recovered to normal mode Dec 27 13:04:52 xen2 multipathd: 360024e800070ed06000004314a967309: remaining ac tive paths: 1 Dec 27 13:04:52 xen2 xapi: [ info|xen2|1118668 UNIX /var/xapi/xapi|session.login _with_password D:8afb60727358|xapi] Session.create trackid=492e0a0a5edb18140a204 dcf801b0eb0 pool=false uname=root is_local_superuser=true auth_user_sid= parent= trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:04:52 xen2 multipathd: 360024e800070ed06000004314a967309: switch to pa th group #4 Dec 27 13:04:52 xen2 kernel: [17527.512748] sd 7:0:0:1: rdac: array MDS-Spindle0 1, ctlr 1, queueing MODE_SELECT command Dec 27 13:04:53 xen2 xapi: [ info|xen2|1118702 UNIX /var/xapi/xapi|session.logou t D:d0126e6b3bad|xapi] Session.destroy trackid=492e0a0a5edb18140a204dcf801b0eb0 Dec 27 13:04:53 xen2 kernel: [17528.017140] sd 7:0:0:1: rdac: array MDS-Spindle0 1, ctlr 1, MODE_SELECT returned with sense 05/91/36 Dec 27 13:04:53 xen2 kernel: [17528.017153] sd 7:0:0:1: rdac: array MDS-Spindle0 1, ctlr 1, retrying MODE_SELECT command Dec 27 13:04:53 xen2 kernel: [17528.303402] sd 7:0:0:1: rdac: array MDS-Spindle0 1, ctlr 1, MODE_SELECT completed Dec 27 13:04:53 xen2 multipathd: 360024e800070ed06000004314a967309: sdf - rdac c hecker reports path is up Dec 27 13:04:53 xen2 multipathd: Path event for 360024e800070ed06000004314a96730 9, request call of mpathcount Dec 27 13:04:53 xen2 multipathd: 8:80: reinstated Dec 27 13:04:53 xen2 multipathd: 360024e800070ed06000004314a967309: remaining ac tive paths: 2 Dec 27 13:04:54 xen2 xapi: [ info|xen2|1118703 UNIX /var/xapi/xapi|session.login _with_password D:cd022f51a23b|xapi] Session.create trackid=78096f7ecdee23b2dae33 38911791e6f pool=false uname=root is_local_superuser=true auth_user_sid= parent= trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:04:54 xen2 multipathd: 360024e800070ed06000004314a967309: load table [ 0 2936012800 multipath 3 pg_init_retries 50 queue_if_no_path 1 rdac 3 3 round-ro bin 0 1 1 8:128 1000 round-robin 0 1 1 8:32 1000 round-robin 0 2 1 8:80 1000 8:1 76 1000] Dec 27 13:04:54 xen2 multipathd: Path event for 360024e800070ed06000004314a96730 9, request call of mpathcount Dec 27 13:04:54 xen2 multipathd: Path event for 360024e800070ed06000004314a96730 9, request call of mpathcount Dec 27 13:04:54 xen2 kernel: [17528.816240] sd 7:0:0:1: [sdl] Unhandled sense co de Dec 27 13:04:54 xen2 kernel: [17528.816243] sd 7:0:0:1: [sdl] Result: hostbyte=D ID_OK driverbyte=DRIVER_SENSE Dec 27 13:04:54 xen2 kernel: [17528.816245] sd 7:0:0:1: [sdl] Sense Key : Hardwa re Error [current] Dec 27 13:04:54 xen2 kernel: [17528.816248] sd 7:0:0:1: [sdl] <<vendor>> ASC=0x8 4 ASCQ=0x0ASC=0x84 ASCQ=0x0 Dec 27 13:04:54 xen2 kernel: [17528.816252] sd 7:0:0:1: [sdl] CDB: Read(10): 28 00 ae ff ff 80 00 00 08 00 Dec 27 13:04:54 xen2 kernel: [17528.816257] end_request: I/O error, dev sdl, sec tor 2936012672 Dec 27 13:04:54 xen2 kernel: [17528.816274] device-mapper: multipath: Failing pa th 8:176. Dec 27 13:04:54 xen2 kernel: [17528.817236] device-mapper: multipath: Failing pa th 8:128. Dec 27 13:04:54 xen2 kernel: [17528.817315] device-mapper: multipath: Failing pa th 8:32. Dec 27 13:04:54 xen2 xapi: [ info|xen2|1118753 UNIX /var/xapi/xapi|session.logou t D:5eabc2a6317b|xapi] Session.destroy trackid=78096f7ecdee23b2dae3338911791e6f Dec 27 13:04:54 xen2 xapi: [ info|xen2|1118755 UNIX /var/xapi/xapi|session.login _with_password D:1894b4b17ab8|xapi] Session.create trackid=920f0d86a645a2f1f126e d2b0fc12f87 pool=false uname=root is_local_superuser=true auth_user_sid= parent= trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:04:54 xen2 xapi: [ info|xen2|1118776 INET 0.0.0.0:80|session.logout D: 3c2cf4cc964b|xapi] Session.destroy trackid=e7b050fd41937a5158bf0b2ca39b1659 Dec 27 13:04:54 xen2 kernel: [17529.332325] sd 5:0:0:1: [sdf] Unhandled sense co de Dec 27 13:04:54 xen2 kernel: [17529.332329] sd 5:0:0:1: [sdf] Result: hostbyte=D ID_OK driverbyte=DRIVER_SENSE Dec 27 13:04:54 xen2 kernel: [17529.332331] sd 5:0:0:1: [sdf] Sense Key : Hardwa re Error [current] Dec 27 13:04:54 xen2 kernel: [17529.332334] sd 5:0:0:1: [sdf] <<vendor>> ASC=0x8 4 ASCQ=0x0ASC=0x84 ASCQ=0x0 Dec 27 13:04:54 xen2 kernel: [17529.332337] sd 5:0:0:1: [sdf] CDB: Read(10): 28 00 ae ff ff 80 00 00 08 00 Dec 27 13:04:54 xen2 kernel: [17529.332343] end_request: I/O error, dev sdf, sec tor 2936012672 Dec 27 13:04:54 xen2 kernel: [17529.332359] device-mapper: multipath: Failing pa th 8:80. Dec 27 13:04:54 xen2 multipathd: 8:80: mark as failed Dec 27 13:04:54 xen2 multipathd: 360024e800070ed06000004314a967309: remaining ac tive paths: 1 Dec 27 13:04:54 xen2 multipathd: Path event for 360024e800070ed06000004314a96730 9, request call of mpathcount Dec 27 13:04:55 xen2 xapi: [ info|xen2|1118805 UNIX /var/xapi/xapi|session.logou t D:41ef5f113fba|xapi] Session.destroy trackid=920f0d86a645a2f1f126ed2b0fc12f87 Dec 27 13:04:55 xen2 xapi: [ info|xen2|1118806 UNIX /var/xapi/xapi|session.login _with_password D:8b9980e8c1ea|xapi] Session.create trackid=8de1347f37c83b6954ea0 9681335db4e pool=false uname=root is_local_superuser=true auth_user_sid= parent= trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:04:55 xen2 multipathd: 360024e800070ed06000004314a967309: sdc - rdac c hecker reports path is ghost Dec 27 13:04:55 xen2 multipathd: Path event for 360024e800070ed06000004314a96730 9, request call of mpathcount Dec 27 13:04:55 xen2 multipathd: 8:32: reinstated Dec 27 13:04:55 xen2 multipathd: 360024e800070ed06000004314a967309: remaining ac tive paths: 2 Dec 27 13:04:55 xen2 multipathd: 360024e800070ed06000004314a967309: switch to pa th group #2 Dec 27 13:04:55 xen2 multipathd: 8:176: mark as failed Dec 27 13:04:55 xen2 multipathd: 360024e800070ed06000004314a967309: remaining ac tive paths: 1 Dec 27 13:04:55 xen2 multipathd: Path event for 360024e800070ed06000004314a96730 9, request call of mpathcount Dec 27 13:04:55 xen2 kernel: [17529.844849] sd 7:0:0:1: [sdl] Unhandled sense co de Dec 27 13:04:55 xen2 kernel: [17529.844852] sd 7:0:0:1: [sdl] Result: hostbyte=D ID_OK driverbyte=DRIVER_SENSE Dec 27 13:04:55 xen2 kernel: [17529.844855] sd 7:0:0:1: [sdl] Sense Key : Hardwa re Error [current] Dec 27 13:04:55 xen2 kernel: [17529.844857] sd 7:0:0:1: [sdl] <<vendor>> ASC=0x8 4 ASCQ=0x0ASC=0x84 ASCQ=0x0 Dec 27 13:04:55 xen2 kernel: [17529.844861] sd 7:0:0:1: [sdl] CDB: Read(10): 28 00 ae ff ff 80 00 00 08 00 Dec 27 13:04:55 xen2 kernel: [17529.844866] end_request: I/O error, dev sdl, sec tor 2936012672 Dec 27 13:04:55 xen2 kernel: [17529.844879] device-mapper: multipath: Failing pa th 8:176. Dec 27 13:04:55 xen2 kernel: [17530.081711] sd 4:0:0:1: rdac: array MDS-Spindle0 1, ctlr 0, queueing MODE_SELECT command Dec 27 13:04:55 xen2 xapi: [ info|xen2|1118849 UNIX /var/xapi/xapi|session.logou t D:0494e32893e6|xapi] Session.destroy trackid=8de1347f37c83b6954ea09681335db4e Dec 27 13:04:55 xen2 xapi: [ info|xen2|1118850 UNIX /var/xapi/xapi|session.login _with_password D:6baeffeb13e0|xapi] Session.create trackid=b043dd9f380b982f1e8aa 32967a8de2b pool=false uname=root is_local_superuser=true auth_user_sid= parent= trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:04:55 xen2 xapi: [ info|xen2|1118852 INET 0.0.0.0:80|session.logout D: 97bc80b93f75|xapi] Session.destroy trackid=53e2afc20c1a0ab616b48da0c7004030 Dec 27 13:04:55 xen2 kernel: [17530.329388] sd 4:0:0:1: rdac: array MDS-Spindle0 1, ctlr 0, MODE_SELECT completed Dec 27 13:04:55 xen2 xapi: [ info|xen2|1118884 UNIX /var/xapi/xapi|session.logou t D:15d77c89ee0a|xapi] Session.destroy trackid=b043dd9f380b982f1e8aa32967a8de2b Dec 27 13:04:56 xen2 kernel: [17530.841619] sd 4:0:0:1: [sdc] Unhandled sense co de Dec 27 13:04:56 xen2 kernel: [17530.841623] sd 4:0:0:1: [sdc] Result: hostbyte=D ID_OK driverbyte=DRIVER_SENSE Dec 27 13:04:56 xen2 kernel: [17530.841628] sd 4:0:0:1: [sdc] Sense Key : Hardwa re Error [current] Dec 27 13:04:56 xen2 kernel: [17530.841632] sd 4:0:0:1: [sdc] <<vendor>> ASC=0x8 4 ASCQ=0x0ASC=0x84 ASCQ=0x0 Dec 27 13:04:56 xen2 kernel: [17530.841638] sd 4:0:0:1: [sdc] CDB: Read(10): 28 00 ae ff ff 80 00 00 08 00 Dec 27 13:04:56 xen2 kernel: [17530.841648] end_request: I/O error, dev sdc, sec tor 2936012672 Dec 27 13:04:56 xen2 kernel: [17530.841672] device-mapper: multipath: Failing pa th 8:32. Dec 27 13:04:56 xen2 multipathd: 360024e800070ed06000004314a967309: sdi - rdac c hecker reports path is up Dec 27 13:04:56 xen2 multipathd: Path event for 360024e800070ed06000004314a96730 9, request call of mpathcount Dec 27 13:04:56 xen2 multipathd: 8:128: reinstated Dec 27 13:04:56 xen2 multipathd: 360024e800070ed06000004314a967309: remaining ac tive paths: 2 Dec 27 13:04:56 xen2 xapi: [ info|xen2|1118889 UNIX /var/xapi/xapi|session.login _with_password D:f3f2ba7c9b76|xapi] Session.create trackid=1f938a40d5b26b9761de4 6230d454ee5 pool=false uname=root is_local_superuser=true auth_user_sid= parent= trackid=9834f5af41c964e225f24279aefe4e49 -
There you go..
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
We still need to see the logs
[root@xen2 ~]# tail -n 100 messages tail: cannot open `messages' for reading: No such file or directoryThat's not good. That means that something really went wrong. Probably the disk had filled up.
Is there another way to view log entries?
Not if they are not there. That's not a log viewer, that's the logs itself. Are you use that you used the cd command to get into the right directory before trying to view the logs?
I used no cd command. I just tried "cd /var/logs" and was told there was no directory.
We are definitely seeing a problem here
-
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
We still need to see the logs
[root@xen2 ~]# tail -n 100 messages tail: cannot open `messages' for reading: No such file or directoryThat's not good. That means that something really went wrong. Probably the disk had filled up.
Is there another way to view log entries?
Not if they are not there. That's not a log viewer, that's the logs itself. Are you use that you used the cd command to get into the right directory before trying to view the logs?
I used no cd command. I just tried "cd /var/logs" and was told there was no directory.
We are definitely seeing a problem here
Scott, why is the formatting so messed up?
-
@Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
We still need to see the logs
[root@xen2 ~]# tail -n 100 messages tail: cannot open `messages' for reading: No such file or directoryThat's not good. That means that something really went wrong. Probably the disk had filled up.
Is there another way to view log entries?
Not if they are not there. That's not a log viewer, that's the logs itself. Are you use that you used the cd command to get into the right directory before trying to view the logs?
I used no cd command. I just tried "cd /var/logs" and was told there was no directory.
We are definitely seeing a problem here
Scott, why is the formatting so messed up?
No idea, but it's not from ML, that's from his cut and paste. Those spaces are all in his original.
-
Dec 27 13:13:16 xen2 last message repeated 2 times Dec 27 13:13:16 xen2 kernel: [18030.894547] device-mapper: multipath: Failing path 8:128. Dec 27 13:13:16 xen2 kernel: [18030.894934] device-mapper: multipath: Failing path 8:80. Dec 27 13:13:16 xen2 kernel: [18030.895002] device-mapper: multipath: Failing path 8:176. Dec 27 13:13:16 xen2 xapi: [ info|xen2|1151320 UNIX /var/xapi/xapi|session.logout D:9b0536d5f01c|xapi] Session.destroy trackid=e8dec3c23c8d667dfcfbfd772ecabd9e Dec 27 13:13:16 xen2 xapi: [ info|xen2|1151321 UNIX /var/xapi/xapi|session.login_with_password D:72d677979b15|xapi] Session.create trackid=2db03e1031b1b474ab4e3a61a73c4824 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:13:16 xen2 kernel: [18031.409957] sd 4:0:0:1: [sdc] Unhandled sense code Dec 27 13:13:16 xen2 kernel: [18031.409960] sd 4:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Dec 27 13:13:16 xen2 kernel: [18031.409963] sd 4:0:0:1: [sdc] Sense Key : Hardware Error [current] Dec 27 13:13:16 xen2 kernel: [18031.409965] sd 4:0:0:1: [sdc] <<vendor>> ASC=0x84 ASCQ=0x0ASC=0x84 ASCQ=0x0 Dec 27 13:13:16 xen2 kernel: [18031.409969] sd 4:0:0:1: [sdc] CDB: Read(10): 28 00 ae ff ff 80 00 00 08 00 Dec 27 13:13:16 xen2 kernel: [18031.409974] end_request: I/O error, dev sdc, sector 2936012672 Dec 27 13:13:16 xen2 kernel: [18031.409989] device-mapper: multipath: Failing path 8:32. Dec 27 13:13:16 xen2 xapi: [ info|xen2|1151359 INET 0.0.0.0:80|session.logout D:6b5f0e1950d3|xapi] Session.destroy trackid=95373089b3a53ad07d348ea483840e40 Dec 27 13:13:16 xen2 multipathd: 8:32: mark as failed Dec 27 13:13:16 xen2 multipathd: 360024e800070ed06000004314a967309: Entering recovery mode: max_retries=15 Dec 27 13:13:16 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 0 Dec 27 13:13:16 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount Dec 27 13:13:16 xen2 xapi: [ info|xen2|1151365 UNIX /var/xapi/xapi|session.logout D:7808df6ecf1f|xapi] Session.destroy trackid=2db03e1031b1b474ab4e3a61a73c4824 Dec 27 13:13:16 xen2 xapi: [ info|xen2|1151366 UNIX /var/xapi/xapi|session.login_with_password D:107e4867f954|xapi] Session.create trackid=398819a8505258b63db68cd0d33af4b1 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:13:17 xen2 multipathd: 360024e800070ed06000004314a967309: sdi - rdac checker reports path is up Dec 27 13:13:17 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount Dec 27 13:13:17 xen2 multipathd: 8:128: reinstated Dec 27 13:13:17 xen2 multipathd: 360024e800070ed06000004314a967309: queue_if_no_path enabled Dec 27 13:13:17 xen2 multipathd: 360024e800070ed06000004314a967309: Recovered to normal mode Dec 27 13:13:17 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 1 Dec 27 13:13:17 xen2 xapi: [ info|xen2|1151408 UNIX /var/xapi/xapi|session.logout D:96b0ad24c6c1|xapi] Session.destroy trackid=398819a8505258b63db68cd0d33af4b1 Dec 27 13:13:17 xen2 xapi: [ info|xen2|1151409 UNIX /var/xapi/xapi|session.login_with_password D:f87fb7e4fce8|xapi] Session.create trackid=a797b1b90698608ddd5f6919804b5484 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:13:17 xen2 kernel: [18032.440371] sd 6:0:0:1: [sdi] Unhandled sense code Dec 27 13:13:17 xen2 kernel: [18032.440374] sd 6:0:0:1: [sdi] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Dec 27 13:13:17 xen2 kernel: [18032.440376] sd 6:0:0:1: [sdi] Sense Key : Hardware Error [current] Dec 27 13:13:17 xen2 kernel: [18032.440379] sd 6:0:0:1: [sdi] <<vendor>> ASC=0x84 ASCQ=0x0ASC=0x84 ASCQ=0x0 Dec 27 13:13:17 xen2 kernel: [18032.440382] sd 6:0:0:1: [sdi] CDB: Read(10): 28 00 ae ff ff 80 00 00 08 00 Dec 27 13:13:17 xen2 kernel: [18032.440387] end_request: I/O error, dev sdi, sector 2936012672 Dec 27 13:13:17 xen2 kernel: [18032.440404] device-mapper: multipath: Failing path 8:128. Dec 27 13:13:17 xen2 multipathd: 8:128: mark as failed Dec 27 13:13:17 xen2 multipathd: 360024e800070ed06000004314a967309: Entering recovery mode: max_retries=15 Dec 27 13:13:17 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 0 Dec 27 13:13:17 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount Dec 27 13:13:17 xen2 xapi: [ info|xen2|1151447 UNIX /var/xapi/xapi|session.logout D:acca552975a2|xapi] Session.destroy trackid=a797b1b90698608ddd5f6919804b5484 Dec 27 13:13:17 xen2 xapi: [ info|xen2|1151448 UNIX /var/xapi/xapi|session.login_with_password D:5f3e3fccf3d5|xapi] Session.create trackid=53cb15330f2848d21867cbf051db9ed9 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:13:18 xen2 multipathd: 360024e800070ed06000004314a967309: sdl - rdac checker reports path is ghost Dec 27 13:13:18 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount Dec 27 13:13:18 xen2 multipathd: 8:176: reinstated Dec 27 13:13:18 xen2 multipathd: 360024e800070ed06000004314a967309: queue_if_no_path enabled Dec 27 13:13:18 xen2 multipathd: 360024e800070ed06000004314a967309: Recovered to normal mode Dec 27 13:13:18 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 1 Dec 27 13:13:18 xen2 multipathd: 360024e800070ed06000004314a967309: switch to path group #4 Dec 27 13:13:18 xen2 kernel: [18033.217827] sd 7:0:0:1: rdac: array MDS-Spindle01, ctlr 1, queueing MODE_SELECT command Dec 27 13:13:18 xen2 xapi: [ info|xen2|1151482 UNIX /var/xapi/xapi|session.logout D:39dea3d23bb2|xapi] Session.destroy trackid=53cb15330f2848d21867cbf051db9ed9 Dec 27 13:13:18 xen2 xapi: [ info|xen2|1151483 UNIX /var/xapi/xapi|session.login_with_password D:487d0c440a6f|xapi] Session.create trackid=31c6550a7ad5c485e04e1ce80f3588dd pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:13:18 xen2 kernel: [18033.500501] sd 7:0:0:1: rdac: array MDS-Spindle01, ctlr 1, MODE_SELECT completed Dec 27 13:13:19 xen2 xapi: [ info|xen2|1151517 UNIX /var/xapi/xapi|session.logout D:6b32233bc36f|xapi] Session.destroy trackid=31c6550a7ad5c485e04e1ce80f3588dd Dec 27 13:13:19 xen2 kernel: [18034.028373] sd 7:0:0:1: [sdl] Unhandled sense code Dec 27 13:13:19 xen2 kernel: [18034.028376] sd 7:0:0:1: [sdl] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Dec 27 13:13:19 xen2 kernel: [18034.028379] sd 7:0:0:1: [sdl] Sense Key : Hardware Error [current] Dec 27 13:13:19 xen2 kernel: [18034.028381] sd 7:0:0:1: [sdl] <<vendor>> ASC=0x84 ASCQ=0x0ASC=0x84 ASCQ=0x0 Dec 27 13:13:19 xen2 kernel: [18034.028385] sd 7:0:0:1: [sdl] CDB: Read(10): 28 00 ae ff ff 80 00 00 08 00 Dec 27 13:13:19 xen2 kernel: [18034.028390] end_request: I/O error, dev sdl, sector 2936012672 Dec 27 13:13:19 xen2 kernel: [18034.028406] device-mapper: multipath: Failing path 8:176. Dec 27 13:13:19 xen2 xapi: [ info|xen2|1151537 INET 0.0.0.0:80|session.logout D:7e74bbbad814|xapi] Session.destroy trackid=57a035032e821d9289ab2ddc9c56ef23 Dec 27 13:13:19 xen2 multipathd: 8:176: mark as failed Dec 27 13:13:19 xen2 multipathd: 360024e800070ed06000004314a967309: Entering recovery mode: max_retries=15 Dec 27 13:13:19 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 0 Dec 27 13:13:19 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount Dec 27 13:13:19 xen2 xapi: [ info|xen2|1151540 UNIX /var/xapi/xapi|session.login_with_password D:d750b0705d7f|xapi] Session.create trackid=86b83edaef195351a3b0fab7cc6cc21e pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:13:20 xen2 xapi: [ info|xen2|1151585 UNIX /var/xapi/xapi|session.logout D:3e57cedfef7a|xapi] Session.destroy trackid=86b83edaef195351a3b0fab7cc6cc21e Dec 27 13:13:20 xen2 multipathd: 360024e800070ed06000004314a967309: sdf - rdac checker reports path is up Dec 27 13:13:20 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount Dec 27 13:13:20 xen2 multipathd: 8:80: reinstated Dec 27 13:13:20 xen2 multipathd: 360024e800070ed06000004314a967309: queue_if_no_path enabled Dec 27 13:13:20 xen2 multipathd: 360024e800070ed06000004314a967309: Recovered to normal mode Dec 27 13:13:20 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 1 Dec 27 13:13:20 xen2 multipathd: 360024e800070ed06000004314a967309: switch to path group #3 Dec 27 13:13:20 xen2 xapi: [ info|xen2|1151605 UNIX /var/xapi/xapi|session.login_with_password D:1379b8d2955b|xapi] Session.create trackid=3fa7db291a48d0f98ea6e364b1dccd5b pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:13:20 xen2 xapi: [ info|xen2|1151611 INET 0.0.0.0:80|session.logout D:e6afd1da2cf0|xapi] Session.destroy trackid=a246f22117a6ddf72bd492a7e8505ddd Dec 27 13:13:21 xen2 kernel: [18035.743234] sd 5:0:0:1: [sdf] Unhandled sense code Dec 27 13:13:21 xen2 kernel: [18035.743237] sd 5:0:0:1: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Dec 27 13:13:21 xen2 kernel: [18035.743240] sd 5:0:0:1: [sdf] Sense Key : Hardware Error [current] Dec 27 13:13:21 xen2 kernel: [18035.743242] sd 5:0:0:1: [sdf] <<vendor>> ASC=0x84 ASCQ=0x0ASC=0x84 ASCQ=0x0 Dec 27 13:13:21 xen2 kernel: [18035.743246] sd 5:0:0:1: [sdf] CDB: Read(10): 28 00 ae ff ff 80 00 00 08 00 Dec 27 13:13:21 xen2 kernel: [18035.743251] end_request: I/O error, dev sdf, sector 2936012672 Dec 27 13:13:21 xen2 kernel: [18035.743266] device-mapper: multipath: Failing path 8:80. Dec 27 13:13:21 xen2 multipathd: 8:80: mark as failed Dec 27 13:13:21 xen2 multipathd: 360024e800070ed06000004314a967309: Entering recovery mode: max_retries=15 Dec 27 13:13:21 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 0 Dec 27 13:13:21 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount Dec 27 13:13:21 xen2 xapi: [ info|xen2|1151653 UNIX /var/xapi/xapi|session.logout D:57acea842a91|xapi] Session.destroy trackid=3fa7db291a48d0f98ea6e364b1dccd5b Dec 27 13:13:21 xen2 xapi: [ info|xen2|1151654 UNIX /var/xapi/xapi|session.login_with_password D:e547785afb43|xapi] Session.create trackid=8fd799a70bdc34fb2a4d8bd676873ede pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 Dec 27 13:13:21 xen2 multipathd: 360024e800070ed06000004314a967309: sdc - rdac checker reports path is ghost Dec 27 13:13:21 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount Dec 27 13:13:21 xen2 multipathd: 8:32: reinstated Dec 27 13:13:21 xen2 multipathd: 360024e800070ed06000004314a967309: queue_if_no_path enabled Dec 27 13:13:21 xen2 multipathd: 360024e800070ed06000004314a967309: Recovered to normal mode Dec 27 13:13:21 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 1 Dec 27 13:13:21 xen2 multipathd: 360024e800070ed06000004314a967309: switch to path group #2 Dec 27 13:13:21 xen2 kernel: [18036.256313] sd 4:0:0:1: rdac: array MDS-Spindle01, ctlr 0, queueing MODE_SELECT command Dec 27 13:13:21 xen2 kernel: [18036.525838] sd 4:0:0:1: rdac: array MDS-Spindle01, ctlr 0, MODE_SELECT completed Dec 27 13:13:22 xen2 xapi: [ info|xen2|1151678 UNIX /var/xapi/xapi|session.logout D:34466f735dad|xapi] Session.destroy trackid=8fd799a70bdc34fb2a4d8bd676873ede Dec 27 13:13:22 xen2 xapi: [ info|xen2|1151679 UNIX /var/xapi/xapi|session.login_with_password D:cd7b037c4106|xapi] Session.create trackid=f258157e7fefe65077464bbd3c912521 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49 -
Better?
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
Better?
Yup, thanks.
-
We need to see the Xen logs, too. Those are the logs with "xen" in the name.
-
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
We need to see the Xen logs, too. Those are the logs with "xen" in the name.
I can't upload the .gz to here. Can I open the contents in wordpad? Sorry if I missed something.
-
No current logs at all?
-
It ate my text. How do I display the logs to you? Sorry. A lot of new knowledge in a short period of time, combined with stress.
-
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
No current logs at all?
-
Sorry, ate my text again. Unless I should be looking somewhere else...
-
Is there any way to tell why the hosts can't even attach to their own local storage, or why it was unplugged in the first place?
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
Sorry, ate my text again. Unless I should be looking somewhere else...
Get us the SMlog tail...
tail -n 100 SMlog
