ML
    • Recent
    • Categories
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!

    IT Discussion
    xenserver xenserver 6.2 iscsi san
    10
    243
    38.9k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • scottalanmillerS
      scottalanmiller
      last edited by

      Then use a command like this to get just the end of the main log file:

      tail -n 100 messages
      C 1 Reply Last reply Reply Quote 0
      • C
        CitrixNewbJD @scottalanmiller
        last edited by

        @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

        @CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

        @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

        @CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

        @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

        I think that we need to look at the logs. Do you have access to get onto the machines themselves via SSH?

        I'm logged into Xen2 with SSH now.

        Great. Let's get as many logs as we can. Likely what we need to know is going to be in there. The logs are under /var/log

        Forgive me for the basic knowledge, how do I access logs from here? The first time I did anything with the Xen CLI was yesterday.

        This is a standard BASH environment. Nothing "Xen" needs to be known.

        To get to the log directory:

        cd /var/log
ls

        That will list out the available logs.

        Yeah, I've heard of BASH, but I've never used it. I've been fortunate enough (or unfortunate, as the case seems to be today) to have been in primarily GUI environments. Here's the screenshots of the log listings.

        0_1482855766667_Screenshot 2016-12-27 10.21.35.png 0_1482855772096_Screenshot 2016-12-27 10.21.38.png

        1 Reply Last reply Reply Quote 0
        • C
          CitrixNewbJD @scottalanmiller
          last edited by scottalanmiller

          @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

          Then use a command like this to get just the end of the main log file:

          tail -n 100 messages

          Dec 27 10:23:04 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: sdh - rdac checker reports path is up
Dec 27 10:23:04 xen1 multipathd: Path event for 360024e800070ed06000007f54dba7bfb, request call of mpathcount
Dec 27 10:23:04 xen1 multipathd: 8:112: reinstated
Dec 27 10:23:04 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: queue_if_no_path enabled
Dec 27 10:23:04 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: Recovered to normal mode
Dec 27 10:23:04 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: remaining active paths: 1
Dec 27 10:23:04 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: switch to path group #2
Dec 27 10:23:05 xen1 kernel: [ 6268.202264] sd 5:0:0:3: [sdh] Unhandled sense code
Dec 27 10:23:05 xen1 kernel: [ 6268.202267] sd 5:0:0:3: [sdh] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Dec 27 10:23:05 xen1 kernel: [ 6268.202269] sd 5:0:0:3: [sdh] Sense Key : Hardware Error [current]
Dec 27 10:23:05 xen1 kernel: [ 6268.202272] sd 5:0:0:3: [sdh] <<vendor>> ASC=0x84 ASCQ=0x0ASC=0x84 ASCQ=0x0
Dec 27 10:23:05 xen1 kernel: [ 6268.202275] sd 5:0:0:3: [sdh] CDB: Read(16): 88 00 00 00 00 01 6e 59 1f 80 00 00 00 08 00 00
Dec 27 10:23:05 xen1 kernel: [ 6268.202282] end_request: I/O error, dev sdh, sector 6146301824
Dec 27 10:23:05 xen1 kernel: [ 6268.202298] device-mapper: multipath: Failing path 8:112.
Dec 27 10:23:05 xen1 multipathd: 8:112: mark as failed
Dec 27 10:23:05 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: Entering recovery mode: max_retries=15
Dec 27 10:23:05 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: remaining active paths: 0
Dec 27 10:23:05 xen1 multipathd: Path event for 360024e800070ed06000007f54dba7bfb, request call of mpathcount
Dec 27 10:23:05 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: sde - rdac checker reports path is ghost
Dec 27 10:23:05 xen1 multipathd: Path event for 360024e800070ed06000007f54dba7bfb, request call of mpathcount
Dec 27 10:23:05 xen1 multipathd: 8:64: reinstated
Dec 27 10:23:05 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: queue_if_no_path enabled
Dec 27 10:23:05 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: Recovered to normal mode
Dec 27 10:23:05 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: remaining active paths: 1
Dec 27 10:23:05 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: switch to path group #1
Dec 27 10:23:05 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: sdn - rdac checker reports path is ghost
Dec 27 10:23:05 xen1 multipathd: Path event for 360024e800070ed06000007f54dba7bfb, request call of mpathcount
Dec 27 10:23:05 xen1 multipathd: 8:208: reinstated
Dec 27 10:23:05 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: remaining active paths: 2
Dec 27 10:23:05 xen1 kernel: [ 6268.701997] sd 4:0:0:3: rdac: array MDS-Spindle01, ctlr 0, queueing MODE_SELECT command
Dec 27 10:23:06 xen1 kernel: [ 6268.970907] sd 4:0:0:3: rdac: array MDS-Spindle01, ctlr 0, MODE_SELECT completed
Dec 27 10:23:06 xen1 xapi: [ info|xen1|131517 UNIX /var/xapi/xapi|session.login_with_password D:e266c465d84c|xapi] Session.create trackid=cc8a9f3a4e457a8654c34f1b256320d4 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 10:23:06 xen1 kernel: [ 6269.499770] sd 7:0:0:3: [sdn] Unhandled sense code
Dec 27 10:23:06 xen1 kernel: [ 6269.499773] sd 7:0:0:3: [sdn] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Dec 27 10:23:06 xen1 kernel: [ 6269.499775] sd 7:0:0:3: [sdn] Sense Key : Hardware Error [current]
Dec 27 10:23:06 xen1 kernel: [ 6269.499778] sd 7:0:0:3: [sdn] <<vendor>> ASC=0x84 ASCQ=0x0ASC=0x84 ASCQ=0x0
Dec 27 10:23:06 xen1 kernel: [ 6269.499781] sd 7:0:0:3: [sdn] CDB: Read(16): 88 00 00 00 00 01 6e 59 1f 80 00 00 00 08 00 00
Dec 27 10:23:06 xen1 kernel: [ 6269.499788] end_request: I/O error, dev sdn, sector 6146301824
Dec 27 10:23:06 xen1 kernel: [ 6269.499805] device-mapper: multipath: Failing path 8:208.
Dec 27 10:23:06 xen1 multipathd: 8:208: mark as failed
Dec 27 10:23:06 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: remaining active paths: 1
Dec 27 10:23:06 xen1 multipathd: Path event for 360024e800070ed06000007f54dba7bfb, request call of mpathcount
Dec 27 10:23:07 xen1 kernel: [ 6270.032558] sd 4:0:0:3: [sde] Unhandled sense code
Dec 27 10:23:07 xen1 kernel: [ 6270.032562] sd 4:0:0:3: [sde] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Dec 27 10:23:07 xen1 kernel: [ 6270.032566] sd 4:0:0:3: [sde] Sense Key : Hardware Error [current]
Dec 27 10:23:07 xen1 kernel: [ 6270.032571] sd 4:0:0:3: [sde] <<vendor>> ASC=0x84 ASCQ=0x0ASC=0x84 ASCQ=0x0
Dec 27 10:23:07 xen1 kernel: [ 6270.032577] sd 4:0:0:3: [sde] CDB: Read(16): 88 00 00 00 00 01 6e 59 1f 80 00 00 00 08 00 00
Dec 27 10:23:07 xen1 kernel: [ 6270.032589] end_request: I/O error, dev sde, sector 6146301824
Dec 27 10:23:07 xen1 kernel: [ 6270.032610] device-mapper: multipath: Failing path 8:64.
Dec 27 10:23:07 xen1 xapi: [ info|xen1|131545 UNIX /var/xapi/xapi|session.login_with_password D:d69e66c86e06|xapi] Session.create trackid=39eeefd838b77788b3ccc71c51581cd6 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 10:23:07 xen1 multipathd: 8:64: mark as failed
Dec 27 10:23:07 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: Entering recovery mode: max_retries=15
Dec 27 10:23:07 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: remaining active paths: 0
Dec 27 10:23:07 xen1 multipathd: Path event for 360024e800070ed06000007f54dba7bfb, request call of mpathcount
Dec 27 10:23:08 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: Entering recovery mode: max_retries=15
Dec 27 10:23:08 xen1 ntpd[6793]: sendto(208.75.88.4) (fd=18): Invalid argument
Dec 27 10:23:08 xen1 xapi: [ info|xen1|131573 UNIX /var/xapi/xapi|session.login_with_password D:b7fd4c2e410c|xapi] Session.create trackid=4706ed452324a154ea9a85d6841d8507 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 10:23:09 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: sdk - rdac checker reports path is ghost
Dec 27 10:23:09 xen1 multipathd: Path event for 360024e800070ed06000007f54dba7bfb, request call of mpathcount
Dec 27 10:23:09 xen1 multipathd: 8:160: reinstated
Dec 27 10:23:09 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: queue_if_no_path enabled
Dec 27 10:23:09 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: Recovered to normal mode
Dec 27 10:23:09 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: remaining active paths: 1
Dec 27 10:23:09 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: switch to path group #3
Dec 27 10:23:09 xen1 kernel: [ 6271.802004] sd 6:0:0:3: rdac: array MDS-Spindle01, ctlr 1, queueing MODE_SELECT command
Dec 27 10:23:09 xen1 xapi: [ info|xen1|131597 UNIX /var/xapi/xapi|session.login_with_password D:f77b1779ddc8|xapi] Session.create trackid=85bc3f2b6b5d775d09f80b9eba992380 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 10:23:09 xen1 kernel: [ 6272.077698] sd 6:0:0:3: rdac: array MDS-Spindle01, ctlr 1, MODE_SELECT completed
Dec 27 10:23:09 xen1 kernel: [ 6272.597687] sd 6:0:0:3: [sdk] Unhandled sense code
Dec 27 10:23:09 xen1 kernel: [ 6272.597690] sd 6:0:0:3: [sdk] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Dec 27 10:23:09 xen1 kernel: [ 6272.597692] sd 6:0:0:3: [sdk] Sense Key : Hardware Error [current]
Dec 27 10:23:09 xen1 kernel: [ 6272.597695] sd 6:0:0:3: [sdk] <<vendor>> ASC=0x84 ASCQ=0x0ASC=0x84 ASCQ=0x0
Dec 27 10:23:09 xen1 kernel: [ 6272.597699] sd 6:0:0:3: [sdk] CDB: Read(16): 88 00 00 00 00 01 6e 59 1f 80 00 00 00 08 00 00
Dec 27 10:23:09 xen1 kernel: [ 6272.597705] end_request: I/O error, dev sdk, sector 6146301824
Dec 27 10:23:09 xen1 kernel: [ 6272.597722] device-mapper: multipath: Failing path 8:160.
Dec 27 10:23:09 xen1 multipathd: 8:160: mark as failed
Dec 27 10:23:09 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: Entering recovery mode: max_retries=15
Dec 27 10:23:09 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: remaining active paths: 0
Dec 27 10:23:09 xen1 multipathd: Path event for 360024e800070ed06000007f54dba7bfb, request call of mpathcount
Dec 27 10:23:10 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: Entering recovery mode: max_retries=15
Dec 27 10:23:10 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: sdh - rdac checker reports path is up
Dec 27 10:23:10 xen1 multipathd: Path event for 360024e800070ed06000007f54dba7bfb, request call of mpathcount
Dec 27 10:23:10 xen1 multipathd: 8:112: reinstated
Dec 27 10:23:10 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: queue_if_no_path enabled
Dec 27 10:23:10 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: Recovered to normal mode
Dec 27 10:23:10 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: remaining active paths: 1
Dec 27 10:23:10 xen1 xapi: [ info|xen1|131625 UNIX /var/xapi/xapi|session.login_with_password D:9820afdb1736|xapi] Session.create trackid=e1073887bd1d5f6e1a101d7313609e20 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 10:23:10 xen1 xcp-rrdd: [error|xen1|0 monitor|main|rrdd_server] Failed to process plugin: xcp-rrdd-xenpm (Rrdd_server.Plugin.No_update)
Dec 27 10:23:10 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: switch to path group #2
Dec 27 10:23:10 xen1 kernel: [ 6273.763175] sd 5:0:0:3: [sdh] Unhandled sense code
Dec 27 10:23:10 xen1 kernel: [ 6273.763179] sd 5:0:0:3: [sdh] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Dec 27 10:23:10 xen1 kernel: [ 6273.763183] sd 5:0:0:3: [sdh] Sense Key : Hardware Error [current]
Dec 27 10:23:10 xen1 kernel: [ 6273.763187] sd 5:0:0:3: [sdh] <<vendor>> ASC=0x84 ASCQ=0x0ASC=0x84 ASCQ=0x0
Dec 27 10:23:10 xen1 kernel: [ 6273.763194] sd 5:0:0:3: [sdh] CDB: Read(16): 88 00 00 00 00 01 6e 59 1f 80 00 00 00 08 00 00
Dec 27 10:23:10 xen1 kernel: [ 6273.763206] end_request: I/O error, dev sdh, sector 6146301824
Dec 27 10:23:10 xen1 kernel: [ 6273.763228] device-mapper: multipath: Failing path 8:112.
Dec 27 10:23:11 xen1 multipathd: 8:112: mark as failed
Dec 27 10:23:11 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: Entering recovery mode: max_retries=15
Dec 27 10:23:11 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: remaining active paths: 0
Dec 27 10:23:11 xen1 multipathd: Path event for 360024e800070ed06000007f54dba7bfb, request call of mpathcount
Dec 27 10:23:11 xen1 multipathd: 360024e800070ed06000007f54dba7bfb: Entering recovery mode: max_retries=15
          1 Reply Last reply Reply Quote 0
          • scottalanmillerS
            scottalanmiller
            last edited by

            Check how your post was edited. There are "ticks" added before and after your log entry to make it appear as logs rather than as generic text.

            C 1 Reply Last reply Reply Quote 0
            • scottalanmillerS
              scottalanmiller
              last edited by

              I'm seeing device errors on the local hardware in the log.

              Can you include the output of:

              df -h

              and...

              lsblk
              C 1 Reply Last reply Reply Quote 0
              • C
                CitrixNewbJD @scottalanmiller
                last edited by

                @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                Check how your post was edited. There are "ticks" added before and after your log entry to make it appear as logs rather than as generic text.

                Edited

                scottalanmillerS 1 Reply Last reply Reply Quote 0
                • scottalanmillerS
                  scottalanmiller @CitrixNewbJD
                  last edited by

                  @CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                  @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                  Check how your post was edited. There are "ticks" added before and after your log entry to make it appear as logs rather than as generic text.

                  Edited

                  It was already fixed. You broke it again 😉 The note was for you to look and see how to make logs appear correctly.

                  C 1 Reply Last reply Reply Quote 0
                  • C
                    CitrixNewbJD @scottalanmiller
                    last edited by scottalanmiller

                    @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                    I'm seeing device errors on the local hardware in the log.

                    Can you include the output of:

                    df -h

                    and...

                    lsblk

                    [root@xen1 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             4.0G  2.8G  986M  75% /
none                  373M   72K  373M   1% /dev/shm
/opt/xensource/packages/iso/XenCenter.iso
                       52M   52M     0 100% /var/xen/xc-install

                    0_1482856470106_Screenshot 2016-12-27 10.34.23.png

                    1 Reply Last reply Reply Quote 0
                    • C
                      CitrixNewbJD @scottalanmiller
                      last edited by

                      @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                      @CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                      @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                      Check how your post was edited. There are "ticks" added before and after your log entry to make it appear as logs rather than as generic text.

                      Edited

                      It was already fixed. You broke it again 😉 The note was for you to look and see how to make logs appear correctly.

                      Doh!

                      1 Reply Last reply Reply Quote 0
                      • scottalanmillerS
                        scottalanmiller
                        last edited by

                        Okay, so the mounted filesystem has enough space. That looks good.

                        C 1 Reply Last reply Reply Quote 0
                        • NerdyDadN
                          NerdyDad
                          last edited by

                          Once this crisis is over, I recommend going over this when you have free time.

                          https://mangolassi.it/topic/7825/sam-learning-linux-system-administration

                          Its not complete but its a really good start.

                          C 1 Reply Last reply Reply Quote 4
                          • C
                            CitrixNewbJD @NerdyDad
                            last edited by

                            @NerdyDad said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                            Once this crisis is over, I recommend going over this when you have free time.

                            https://mangolassi.it/topic/7825/sam-learning-linux-system-administration

                            Its not complete but its a really good start.

                            Learning Linux has always been something I've wanted to do, but have never been able to find the time to do so in a work environment.

                            NerdyDadN 1 Reply Last reply Reply Quote 1
                            • NerdyDadN
                              NerdyDad @CitrixNewbJD
                              last edited by

                              @CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                              @NerdyDad said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                              Once this crisis is over, I recommend going over this when you have free time.

                              https://mangolassi.it/topic/7825/sam-learning-linux-system-administration

                              Its not complete but its a really good start.

                              Learning Linux has always been something I've wanted to do, but have never been able to find the time to do so in a work environment.

                              I totally understand. Either on your laptop or as another VM in XS, I would setup Linux. Then just take a topic or 2 a day and learn a new skill. After a while, you will be building one skill on top of another.

                              1 Reply Last reply Reply Quote 0
                              • C
                                CitrixNewbJD @scottalanmiller
                                last edited by

                                Forgive me and my anxiousness, but are you waiting on an action from me, or are you in a paused mode?

                                1 Reply Last reply Reply Quote 0
                                • scottalanmillerS
                                  scottalanmiller
                                  last edited by

                                  Let's install that tool if we can...

                                  yum -y install lsblk
                                  C 1 Reply Last reply Reply Quote 0
                                  • C
                                    CitrixNewbJD @scottalanmiller
                                    last edited by

                                    @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                                    Let's install that tool if we can...

                                    yum -y install lsblk

                                    0_1482859744101_Screenshot 2016-12-27 11.28.56.png

                                    1 Reply Last reply Reply Quote 0
                                    • scottalanmillerS
                                      scottalanmiller
                                      last edited by

                                      That's not promising.

                                      C 1 Reply Last reply Reply Quote 0
                                      • C
                                        CitrixNewbJD @scottalanmiller
                                        last edited by

                                        @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                                        That's not promising.

                                        This is still all on Xen1. Do you want me to try anything on Xen2, the pool master?

                                        DashrenderD scottalanmillerS 2 Replies Last reply Reply Quote 0
                                        • DashrenderD
                                          Dashrender @CitrixNewbJD
                                          last edited by

                                          @CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                                          @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                                          That's not promising.

                                          This is still all on Xen1. Do you want me to try anything on Xen2, the pool master?

                                          did you ever find out if Xen2 is on the network? able to ping the SAN?

                                          C 1 Reply Last reply Reply Quote 0
                                          • scottalanmillerS
                                            scottalanmiller @CitrixNewbJD
                                            last edited by

                                            @CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                                            @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                                            That's not promising.

                                            This is still all on Xen1. Do you want me to try anything on Xen2, the pool master?

                                            Do this on there, too..

                                            df -h
                                            C 1 Reply Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 3
                                            • 4
                                            • 5
                                            • 12
                                            • 13
                                            • 2 / 13
                                            • First post
                                              Last post