Posts made by CitrixNewbJD

CitrixNewbJD

So, when looking for places to turn off AD integration, I see this...

0_1482873847039_Screenshot 2016-12-27 15.23.59.png

CitrixNewbJD

Frank, can we speak on the phone for a minute so I can be sure I can intelligently talk to the guy when demanding his money?

CitrixNewbJD

@momurda

I've been using the root authentication for everything.

CitrixNewbJD

@momurda

Having been through this once before, and learning the hard way, I do normally have a physical DC. Despite my warnings, because I know that we do not currently have one here, I was told to bring it all down. And here we are. We do not have a physical DC.

CitrixNewbJD

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@momurda said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

Who is spindleinc\mgould? Is your Xenserver setup with AD integration? it looks like this is the case. Is mgould a disabled account in AD(this seems to be the guy who had your job, maybe). If so, that is a huge issue.
Is there a reason you have 5 ISO libraries?

mgould is still with us. I don't know if it was set up with AD integration, but I would assume so based on this. It's not a disabled account, but we can't get to AD.

That's not good. AD was built on top of a system that was merged with all shared dependencies... so AD needs Xen which needs the PowerVault, which needs Xen which needs AD. So the circular issue potentially? There is no AD to fail over to if anything goes wrong with the Xen cluster has no redundancy. No matter how many copies run on XenServer here, there is only one disk array holding it all.

Does this mean it's irreparable?

CitrixNewbJD

@momurda said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

Who is spindleinc\mgould? Is your Xenserver setup with AD integration? it looks like this is the case. Is mgould a disabled account in AD(this seems to be the guy who had your job, maybe). If so, that is a huge issue.
Is there a reason you have 5 ISO libraries?

mgould is still with us. I don't know if it was set up with AD integration, but I would assume so based on this. It's not a disabled account, but we can't get to AD.

I have no idea why we have 5 ISO libraries.

CitrixNewbJD

@scottalanmiller

[root@xen2 log]# xe pbd-list
uuid ( RO)                  : 360fee57-907f-2c6a-e15d-166412f7bd7e
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): e58b6f8a-225d-2a83-642d-0610812efbea
         device-config (MRO): multihomelist: 192.168.130.101:3260,192.168.130.102:3260,192.168.130.103:3260,192.168.130.104:3260; SCSIid: 360024e800070ed06000004314a967309; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800070ed06000000004a61f4c6; port: 3260; target: 192.168.130.101
    currently-attached ( RO): false


uuid ( RO)                  : c4ee5bb1-f824-52c4-dc09-c2c50034801e
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): e8c1da6c-9016-fa1d-b357-e1bc070b4d2b
         device-config (MRO): multihomelist: 192.168.130.101:3260,192.168.130.102:3260,192.168.130.103:3260,192.168.130.104:3260; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800070ed06000000004a61f4c6; port: 3260; SCSIid: 360024e800070ed06000007f54dba7bfb; target: 192.168.130.101
    currently-attached ( RO): false


uuid ( RO)                  : 5dedd15d-45c6-c38d-7ed1-0eb8bd7a4620
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): d457d592-56c2-73e0-379a-26a55be27538
         device-config (MRO): iso_path: /XenISO; location: //192.168.1.230/Zipfiles; type: cifs; username: spindleinc\mgould; cifspassword_secret: c74bebaa-bbb0-e2d8-9a19-e11433375cd4
    currently-attached ( RO): false


uuid ( RO)                  : 5bd81e5b-895e-d385-bd28-da4a4b2603f8
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): 169601bb-0d24-006a-a02f-e6ccb65a560f
         device-config (MRO): device: /dev/sdb4
    currently-attached ( RO): false


uuid ( RO)                  : ebf92343-b5b1-d530-ebd8-235365c3d961
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): ff6303ba-0852-6f9c-b59f-ddb2cbc4d07b
         device-config (MRO): multihomelist: 192.168.130.108:3260,192.168.130.105:3260,192.168.130.107:3260,192.168.130.106:3260; SCSIid: 360024e800054baef0000ddd550c85082; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800054baef0000000050c83cdb; port: 3260; target: 192.168.130.108
    currently-attached ( RO): false


uuid ( RO)                  : 0f241fac-bf50-5141-69aa-7c31ac0fb406
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): ec1432db-9f89-2a04-987e-263f0d30c66f
         device-config (MRO): iso_path: /Software/Microsoft/Windows/Server/2012; location: //192.168.1.100/files; type: cifs; username: spindleinc\mgould; cifspassword_secret: 77701771-8d4e-3de7-a562-90542842828a
    currently-attached ( RO): false


uuid ( RO)                  : 6fd14ddd-9595-88fe-592f-1d289d06ebbc
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): 7e001a4b-4f2d-15fe-4e7a-653062b4cdcd
         device-config (MRO): location: /dev/xapi/cd
    currently-attached ( RO): false


uuid ( RO)                  : e97a8dbc-eec8-dcde-63ef-fe0f350ed73d
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): 4e97fda1-40cb-8a4a-baba-184a9e81fee5
         device-config (MRO): location: /opt/xensource/packages/iso; legacy_mode: true
    currently-attached ( RO): false


uuid ( RO)                  : 7ab79e27-cbe1-04b0-4c14-f8e89c200411
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): 4e97fda1-40cb-8a4a-baba-184a9e81fee5
         device-config (MRO): location: /opt/xensource/packages/iso; legacy_mode: true
    currently-attached ( RO): false


uuid ( RO)                  : cacb5cd4-3894-8fbc-e3ce-6b342c11df95
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): e8c1da6c-9016-fa1d-b357-e1bc070b4d2b
         device-config (MRO): multihomelist: 192.168.130.101:3260,192.168.130.102:3260,192.168.130.103:3260,192.168.130.104:3260; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800070ed06000000004a61f4c6; port: 3260; SCSIid: 360024e800070ed06000007f54dba7bfb; target: 192.168.130.101
    currently-attached ( RO): false


uuid ( RO)                  : 09b657e0-974e-7a9e-99f8-425e8da32c7b
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): dbdb8c1c-30b6-5e24-991d-7f42c452fd07
         device-config (MRO): multihomelist: 192.168.130.101:3260,192.168.130.102:3260,192.168.130.103:3260,192.168.130.104:3260; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800070ed06000000004a61f4c6; port: 3260; SCSIid: 360024e80007b786a000004134a9670f3; target: 192.168.130.101
    currently-attached ( RO): false


uuid ( RO)                  : 38ac4cf3-15b4-b319-2fc7-44eff01b93f6
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): 49447156-88bc-e5cf-7578-c0a5775d6dc7
         device-config (MRO): device: /dev/disk/by-id/scsi-360024e8072e05b0011fe4fc707bd94eb-part4
    currently-attached ( RO): false


uuid ( RO)                  : 8ffccb24-5c1e-27a9-879d-01e89b419fca
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): 85d27d5d-19cd-df91-62e2-f64cfb7153cf
         device-config (MRO): location: /dev/xapi/block
    currently-attached ( RO): false


uuid ( RO)                  : 6d49b0e1-5d6a-9779-1f23-5c4414121518
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): 5cdb03b4-746d-4ee1-8a45-3b3a33895e69
         device-config (MRO): type: cifs; location: //192.168.1.230/zipfiles; cifspassword_secret: 25876439-7bf0-75be-f977-17ce22ec3536; username: jason; iso_path: /microsoft
    currently-attached ( RO): false


uuid ( RO)                  : 24695844-f828-d80e-a002-08107734c527
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): e58b6f8a-225d-2a83-642d-0610812efbea
         device-config (MRO): multihomelist: 192.168.130.101:3260,192.168.130.102:3260,192.168.130.103:3260,192.168.130.104:3260; SCSIid: 360024e800070ed06000004314a967309; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800070ed06000000004a61f4c6; port: 3260; target: 192.168.130.101
    currently-attached ( RO): false


uuid ( RO)                  : 80f89268-eb01-5460-5ed9-e65b8fe62875
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): ff6303ba-0852-6f9c-b59f-ddb2cbc4d07b
         device-config (MRO): multihomelist: 192.168.130.108:3260,192.168.130.107:3260,192.168.130.105:3260,192.168.130.106:3260; SCSIid: 360024e800054baef0000ddd550c85082; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800054baef0000000050c83cdb; port: 3260; target: 192.168.130.108
    currently-attached ( RO): false


uuid ( RO)                  : 965ead53-f00f-158a-db6d-84865dcdb38d
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): d457d592-56c2-73e0-379a-26a55be27538
         device-config (MRO): iso_path: /XenISO; location: //192.168.1.230/Zipfiles; type: cifs; username: spindleinc\mgould; cifspassword_secret: a44f6bfd-dd64-768d-56e6-cb1c002c5d8c
    currently-attached ( RO): false


uuid ( RO)                  : dbef4921-c3f1-2cb8-6548-d462745bb1ea
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): 5cdb03b4-746d-4ee1-8a45-3b3a33895e69
         device-config (MRO): type: cifs; location: //192.168.1.230/zipfiles; cifspassword_secret: d7287cdb-b3a8-8583-aba4-a2c24d0394d5; username: jason; iso_path: /microsoft
    currently-attached ( RO): false


uuid ( RO)                  : 9b968975-b85e-5369-86f3-d924c9d10871
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): dbdb8c1c-30b6-5e24-991d-7f42c452fd07
         device-config (MRO): multihomelist: 192.168.130.101:3260,192.168.130.102:3260,192.168.130.103:3260,192.168.130.104:3260; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800070ed06000000004a61f4c6; port: 3260; SCSIid: 360024e80007b786a000004134a9670f3; target: 192.168.130.101
    currently-attached ( RO): false


uuid ( RO)                  : c15826f2-0f0b-6fcf-87ab-36b90754368f
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): ba8349a9-87d5-19b0-36b6-c12f4e5b40a9
         device-config (MRO): location: /dev/xapi/cd
    currently-attached ( RO): false


uuid ( RO)                  : 65de2fdc-49dd-c0a4-b397-c44e42e6b998
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): b03b52b1-e403-1649-9595-8f7c1b442896
         device-config (MRO): location: /dev/xapi/block
    currently-attached ( RO): false


uuid ( RO)                  : b602d816-38fc-d5f8-6880-943a3ba824d3
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): ec1432db-9f89-2a04-987e-263f0d30c66f
         device-config (MRO): iso_path: /Software/Microsoft/Windows/Server/2012; location: //192.168.1.100/files; type: cifs; username: spindleinc\mgould; cifspassword_secret: 8ca82529-cbac-4818-db7e-ccdcbb85c193
    currently-attached ( RO): false

CitrixNewbJD

@scottalanmiller

[root@xen2 log]# tail -n 100 xen
tail: error reading `xen': Is a directory

CitrixNewbJD

@scottalanmiller

[root@xen2 log]# tail -n 100 xensource.log
Dec 27 14:31:16 xen2 xapi: [debug|xen2|1450823 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:82887bba0b96|api_effect] PBD.remove_from_other_config
Dec 27 14:31:16 xen2 xapi: [debug|xen2|1450825 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:09e2aa2687dd|api_effect] PBD.remove_from_other_config
Dec 27 14:31:16 xen2 xapi: [debug|xen2|1450827 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:13a7f9b859e0|api_effect] PBD.add_to_other_config
Dec 27 14:31:16 xen2 xapi: [debug|xen2|1450829 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:3270fba588b8|api_effect] PBD.add_to_other_config
Dec 27 14:31:16 xen2 xapi: [ info|xen2|1450838 UNIX /var/xapi/xapi|session.logout D:2ad4d14e2a85|xapi] Session.destroy trackid=402d2a79fc43297a8dc25f7fa36844de
Dec 27 14:31:16 xen2 xapi: [ info|xen2|1450839 UNIX /var/xapi/xapi|session.login_with_password D:82f44b61ad08|xapi] Session.create trackid=9b5f5e4e28b67bce328bb7a29f8c6e61 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 14:31:16 xen2 xapi: [debug|xen2|1450839 UNIX /var/xapi/xapi|session.login_with_password D:82f44b61ad08|mscgen] xapi=>xapi [label="(XML)"];
Dec 27 14:31:16 xen2 xapi: [debug|xen2|1450840 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:df5edc881477 created by task D:82f44b61ad08
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450858 INET 0.0.0.0:80|dispatch:PBD.remove_from_other_config D:9d9efd457fb7|api_effect] PBD.remove_from_other_config
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450860 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:04c9188b6703|api_effect] PBD.remove_from_other_config
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450861 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:49cf176e5e46|api_effect] PBD.remove_from_other_config
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450863 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:aeb76c4c5540|api_effect] PBD.add_to_other_config
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450865 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:400507c7a850|api_effect] PBD.add_to_other_config
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450862 INET 0.0.0.0:80|dispatch:PBD.remove_from_other_config D:7f07845eed3f|api_effect] PBD.remove_from_other_config
Dec 27 14:31:17 xen2 xapi: [ info|xen2|1450875 UNIX /var/xapi/xapi|session.logout D:32207e9ca5af|xapi] Session.destroy trackid=9b5f5e4e28b67bce328bb7a29f8c6e61
Dec 27 14:31:17 xen2 xapi: [ info|xen2|1450876 UNIX /var/xapi/xapi|session.login_with_password D:266336f0c5ca|xapi] Session.create trackid=cbe2ea2161a008b31c123bba3d3088f6 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450876 UNIX /var/xapi/xapi|session.login_with_password D:266336f0c5ca|mscgen] xapi=>xapi [label="(XML)"];
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450877 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:a754d9534632 created by task D:266336f0c5ca
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450885 INET 0.0.0.0:80|dispatch:PBD.add_to_other_config D:5165cfdc6eb0|api_effect] PBD.add_to_other_config
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450897 INET 0.0.0.0:80|dispatch:PBD.add_to_other_config D:f540965ad467|api_effect] PBD.add_to_other_config
Dec 27 14:31:17 xen2 xapi: [ info|xen2|1450910 INET 0.0.0.0:80|session.logout D:41f9af099127|xapi] Session.destroy trackid=fe675f6a258246bd1b69c3f7a5c3e99d
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450911 INET 0.0.0.0:80||dummytaskhelper] task dispatch:session.get_uuid D:68bea1da5826 created by task D:e5fb02a3454d
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|xapi] Marking SR for PBD.plug (task=OpaqueRef:1a2d6e65-ffde-4e66-1904-557a00063c92)
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|backtrace] Raised at xapi_sr_operations.ml:137.33-72 -> message_forwarding.ml:2950.5-67 -> message_forwarding.ml:283.5-8
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|xapi] Caught exception while OTHER_OPERATION_IN_PROGRESS: [ SR; OpaqueRef:37ddc678-2fc1-67d3-a5c8-c53499df2864; plug ] in message forwarder: marking SR for PBD.plug
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|backtrace] Raised at message_forwarding.ml:286.8-9 -> threadext.ml:20.20-24 -> threadext.ml:20.62-65 -> message_forwarding.ml:149.19-25
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|xapi] PBD.plug locking failed: caught transient failure OTHER_OPERATION_IN_PROGRESS: [ SR; OpaqueRef:37ddc678-2fc1-67d3-a5c8-c53499df2864; plug ]
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|xapi] Waiting for up to 1.695234 seconds before retrying...
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450924 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:d3520cfe0fe4|api_effect] PBD.remove_from_other_config
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450926 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:45e6f1af395a|api_effect] PBD.remove_from_other_config
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450928 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:1657b7fd0b3b|api_effect] PBD.add_to_other_config
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450930 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:7a61e7bc2274|api_effect] PBD.add_to_other_config
Dec 27 14:31:17 xen2 xapi: [ info|xen2|1450939 UNIX /var/xapi/xapi|session.logout D:f26e50a00a23|xapi] Session.destroy trackid=cbe2ea2161a008b31c123bba3d3088f6
Dec 27 14:31:17 xen2 xapi: [ info|xen2|1450940 UNIX /var/xapi/xapi|session.login_with_password D:b7f1535237f8|xapi] Session.create trackid=90b7aa6ac3991339a1bd4f0642b76680 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450940 UNIX /var/xapi/xapi|session.login_with_password D:b7f1535237f8|mscgen] xapi=>xapi [label="(XML)"];
Dec 27 14:31:17 xen2 xapi: [debug|xen2|1450941 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:17f47ccae7b3 created by task D:b7f1535237f8
Dec 27 14:31:18 xen2 xapi: [debug|xen2|1450959 INET 0.0.0.0:80|dispatch:PBD.remove_from_other_config D:e7ca4a8dec5b|api_effect] PBD.remove_from_other_config
Dec 27 14:31:18 xen2 xapi: [debug|xen2|1450961 INET 0.0.0.0:80|dispatch:PBD.remove_from_other_config D:4a97c8a92518|api_effect] PBD.remove_from_other_config
Dec 27 14:31:18 xen2 xapi: [debug|xen2|1450963 INET 0.0.0.0:80|dispatch:PBD.add_to_other_config D:5f0b0e1f91ad|api_effect] PBD.add_to_other_config
Dec 27 14:31:18 xen2 xapi: [ info|xen2|1450970 UNIX /var/xapi/xapi|session.logout D:33f81c0e1f75|xapi] Session.destroy trackid=90b7aa6ac3991339a1bd4f0642b76680
Dec 27 14:31:18 xen2 xapi: [debug|xen2|1450971 INET 0.0.0.0:80|dispatch:PBD.add_to_other_config D:3dc55dedcbbb|api_effect] PBD.add_to_other_config
Dec 27 14:31:18 xen2 xapi: [ info|xen2|1450984 INET 0.0.0.0:80|session.logout D:3f6e5af1e625|xapi] Session.destroy trackid=db210ab1f145ee422323350b21bd74dd
Dec 27 14:31:18 xen2 xapi: [debug|xen2|1450985 INET 0.0.0.0:80||dummytaskhelper] task dispatch:session.get_uuid D:e79b78e7a6a5 created by task D:e91fc341b52a
Dec 27 14:31:18 xen2 xapi: [ info|xen2|1450998 UNIX /var/xapi/xapi|session.login_with_password D:35cee6258606|xapi] Session.create trackid=d10e206dd14c6d49f4fe098a4f0766fb pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 14:31:18 xen2 xapi: [debug|xen2|1450998 UNIX /var/xapi/xapi|session.login_with_password D:35cee6258606|mscgen] xapi=>xapi [label="(XML)"];
Dec 27 14:31:18 xen2 xapi: [debug|xen2|1450999 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:6ba2db342589 created by task D:35cee6258606
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1451017 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:34d31751b3ca|api_effect] PBD.remove_from_other_config
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1451019 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:02dd5072cc69|api_effect] PBD.remove_from_other_config
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1451021 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:a8db6c31efc3|api_effect] PBD.add_to_other_config
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1451023 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:def7f44f171f|api_effect] PBD.add_to_other_config
Dec 27 14:31:19 xen2 xapi: [ info|xen2|1451032 UNIX /var/xapi/xapi|session.logout D:142a6672fcff|xapi] Session.destroy trackid=d10e206dd14c6d49f4fe098a4f0766fb
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|xapi] Marking SR for PBD.plug (task=OpaqueRef:1a2d6e65-ffde-4e66-1904-557a00063c92)
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|backtrace] Raised at xapi_sr_operations.ml:137.33-72 -> message_forwarding.ml:2950.5-67 -> message_forwarding.ml:283.5-8
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|xapi] Caught exception while OTHER_OPERATION_IN_PROGRESS: [ SR; OpaqueRef:37ddc678-2fc1-67d3-a5c8-c53499df2864; plug ] in message forwarder: marking SR for PBD.plug
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|backtrace] Raised at message_forwarding.ml:286.8-9 -> threadext.ml:20.20-24 -> threadext.ml:20.62-65 -> message_forwarding.ml:149.19-25
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|xapi] PBD.plug locking failed: caught transient failure OTHER_OPERATION_IN_PROGRESS: [ SR; OpaqueRef:37ddc678-2fc1-67d3-a5c8-c53499df2864; plug ]
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1277861 UNIX /var/xapi/xapi|PBD.plug R:1a2d6e65ffde|xapi] Waiting for up to 12.666722 seconds before retrying...
Dec 27 14:31:19 xen2 xcp-rrdd: [debug|xen2|0 monitor|main|rrdd_stats] system stats: MemTotal: 762248 KiB; MemFree: 142620 KiB; Buffered: 30580 KiB; Cached: 223716 KiB; SwapTotal: 524280 KiB; SwapFree: 524280 KiB
Dec 27 14:31:19 xen2 xcp-rrdd: [debug|xen2|0 monitor|main|rrdd_stats] Clock drift: -0
Dec 27 14:31:19 xen2 xcp-rrdd: [debug|xen2|0 monitor|main|rrdd_stats] xcp-rrdd stats (n = 1): size: 68164 KiB; rss: 5388 KiB; data: 60788 KiB; stack: 88 KiB
Dec 27 14:31:19 xen2 xcp-rrdd: [debug|xen2|0 monitor|main|rrdd_stats] xapi stats (n = 2): size: 361788 KiB; rss: 73772 KiB; data: 330832 KiB; stack: 220 KiB
Dec 27 14:31:19 xen2 xapi: [ info|xen2|1451033 UNIX /var/xapi/xapi|session.login_with_password D:4619a8559d1e|xapi] Session.create trackid=f662a35431bf0ddd237a3447e7ef4f77 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1451033 UNIX /var/xapi/xapi|session.login_with_password D:4619a8559d1e|mscgen] xapi=>xapi [label="(XML)"];
Dec 27 14:31:19 xen2 xapi: [debug|xen2|1451034 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:c9f8df7140fc created by task D:4619a8559d1e
Dec 27 14:31:19 xen2 xcp-rrdd: [debug|xen2|0 monitor|main|rrdd_stats] xenopsd stats (n = 1): size: 114976 KiB; rss: 4348 KiB; data: 109736 KiB; stack: 88 KiB
Dec 27 14:31:19 xen2 xapi: [debug|xen2|6 dbflush [/var/xapi/state.db]||sql] XML backend [/var/xapi/state.db] -- Write buffer flushed. Time: 0.022087
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1289694 UNIX /var/xapi/xapi|PBD.plug R:48b3b6331cca|xapi] Marking SR for PBD.plug (task=OpaqueRef:48b3b633-1cca-1112-4280-62e0c95bd0e0)
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1289694 UNIX /var/xapi/xapi|PBD.plug R:48b3b6331cca|backtrace] Raised at xapi_sr_operations.ml:137.33-72 -> message_forwarding.ml:2950.5-67 -> message_forwarding.ml:283.5-8
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1289694 UNIX /var/xapi/xapi|PBD.plug R:48b3b6331cca|xapi] Caught exception while OTHER_OPERATION_IN_PROGRESS: [ SR; OpaqueRef:37ddc678-2fc1-67d3-a5c8-c53499df2864; plug ] in message forwarder: marking SR for PBD.plug
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1289694 UNIX /var/xapi/xapi|PBD.plug R:48b3b6331cca|backtrace] Raised at message_forwarding.ml:286.8-9 -> threadext.ml:20.20-24 -> threadext.ml:20.62-65 -> message_forwarding.ml:149.19-25
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1289694 UNIX /var/xapi/xapi|PBD.plug R:48b3b6331cca|xapi] PBD.plug locking failed: caught transient failure OTHER_OPERATION_IN_PROGRESS: [ SR; OpaqueRef:37ddc678-2fc1-67d3-a5c8-c53499df2864; plug ]
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1289694 UNIX /var/xapi/xapi|PBD.plug R:48b3b6331cca|xapi] Waiting for up to 14.806154 seconds before retrying...
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1451058 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:12c10588e48e|api_effect] PBD.remove_from_other_config
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1451059 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:f7ca1fefa964|api_effect] PBD.remove_from_other_config
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1451060 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:c60e0a7885a6|api_effect] PBD.add_to_other_config
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1451061 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:7d41478633a2|api_effect] PBD.add_to_other_config
Dec 27 14:31:20 xen2 xapi: [ info|xen2|1451071 UNIX /var/xapi/xapi|session.logout D:d99c9d2cbad6|xapi] Session.destroy trackid=f662a35431bf0ddd237a3447e7ef4f77
Dec 27 14:31:20 xen2 xapi: [ info|xen2|1451072 UNIX /var/xapi/xapi|session.login_with_password D:99d40118134c|xapi] Session.create trackid=41be80a325ec2708158b61cd0f319b9d pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1451072 UNIX /var/xapi/xapi|session.login_with_password D:99d40118134c|mscgen] xapi=>xapi [label="(XML)"];
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1451073 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:09c20f39baca created by task D:99d40118134c
Dec 27 14:31:20 xen2 xapi: [ info|xen2|1451093 INET 0.0.0.0:80|session.logout D:cc28177ddc28|xapi] Session.destroy trackid=b468f6dc63292261522b234fe5ee5ad2
Dec 27 14:31:20 xen2 xapi: [debug|xen2|1451094 INET 0.0.0.0:80||dummytaskhelper] task dispatch:session.get_uuid D:2d0986801026 created by task D:0761c6bbf357
Dec 27 14:31:21 xen2 xapi: [ info|xen2|1451112 UNIX /var/xapi/xapi|session.logout D:fcd319dfb72c|xapi] Session.destroy trackid=41be80a325ec2708158b61cd0f319b9d
Dec 27 14:31:21 xen2 xapi: [ info|xen2|1451113 UNIX /var/xapi/xapi|session.login_with_password D:218ca8a8a66a|xapi] Session.create trackid=061c385d53b9ad60f9b61a224d40b830 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451113 UNIX /var/xapi/xapi|session.login_with_password D:218ca8a8a66a|mscgen] xapi=>xapi [label="(XML)"];
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451114 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:ea5152d18048 created by task D:218ca8a8a66a
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451132 INET 0.0.0.0:80|dispatch:PBD.remove_from_other_config D:e9897d53a20e|api_effect] PBD.remove_from_other_config
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451134 INET 0.0.0.0:80|dispatch:PBD.remove_from_other_config D:7f7b526dec2e|api_effect] PBD.remove_from_other_config
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451136 INET 0.0.0.0:80|dispatch:PBD.add_to_other_config D:1bfda339ff57|api_effect] PBD.add_to_other_config
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451138 INET 0.0.0.0:80|dispatch:PBD.add_to_other_config D:b5333aacf0f0|api_effect] PBD.add_to_other_config
Dec 27 14:31:21 xen2 xapi: [ info|xen2|1451151 INET 0.0.0.0:80|session.logout D:9e312895529a|xapi] Session.destroy trackid=5c0c47b10d423763811399642ffdfafc
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451152 INET 0.0.0.0:80||dummytaskhelper] task dispatch:session.get_uuid D:d5ccd057b9d5 created by task D:7abf77ad1741
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451157 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:5bd0ee2d790b|api_effect] PBD.remove_from_other_config
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451158 UNIX /var/xapi/xapi|dispatch:PBD.remove_from_other_config D:6d3b73bc7395|api_effect] PBD.remove_from_other_config
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451161 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:eb4ac5803e5a|api_effect] PBD.add_to_other_config
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451162 UNIX /var/xapi/xapi|dispatch:PBD.add_to_other_config D:a07c440a7f09|api_effect] PBD.add_to_other_config
Dec 27 14:31:21 xen2 xapi: [ info|xen2|1451169 UNIX /var/xapi/xapi|session.logout D:0c2a055e01cf|xapi] Session.destroy trackid=061c385d53b9ad60f9b61a224d40b830
Dec 27 14:31:21 xen2 xapi: [ info|xen2|1451173 UNIX /var/xapi/xapi|session.login_with_password D:e4b0fd1684b5|xapi] Session.create trackid=9b647868c555905ed290099de411b7f4 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451173 UNIX /var/xapi/xapi|session.login_with_password D:e4b0fd1684b5|mscgen] xapi=>xapi [label="(XML)"];
Dec 27 14:31:21 xen2 xapi: [debug|xen2|1451175 UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid D:7e21e3a33f60 created by task D:e4b0fd1684b5

CitrixNewbJD

@scottalanmiller

[root@xen2 log]# tail -n 100 xen-dmesg
(XEN)  00000000cf3be000 - 00000000d0000000 (reserved)
(XEN)  00000000e0000000 - 00000000f0000000 (reserved)
(XEN)  00000000fe000000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000001230000000 (usable)
(XEN) Kdump: 64MB (65536kB) at 0x2000000
(XEN) ACPI: RSDP 000F1170, 0024 (r2 DELL  )
(XEN) ACPI: XSDT 000F1274, 009C (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: FACP CF3A3F9C, 00F4 (r3 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: DSDT CF37F000, 3D72 (r1 DELL   PE_SC3          1 INTL 20050624)
(XEN) ACPI: FACS CF3A6000, 0040
(XEN) ACPI: APIC CF3A3478, 015E (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: SPCR CF3A35D8, 0050 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: HPET CF3A362C, 0038 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: DMAR CF3A3668, 01B0 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: MCFG CF3A38C4, 003C (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: WD__ CF3A3904, 0134 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: SLIC CF3A3A3C, 0024 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: ERST CF382EF4, 0270 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: HEST CF383164, 03A8 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: BERT CF382D74, 0030 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: EINJ CF382DA4, 0150 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: SRAT CF3A3BC0, 0370 (r1 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: TCPA CF3A3F34, 0064 (r2 DELL   PE_SC3          1 DELL        1)
(XEN) ACPI: SSDT CF3A7000, 4A24 (r1  INTEL PPM RCM  80000001 INTL 20061109)
(XEN) System RAM: 73715MB (75484196kB)
(XEN) Domain heap initialised DMA width 32 bits
(XEN) Processor #16 7:10 APIC version 21
(XEN) Processor #0 7:10 APIC version 21
(XEN) Processor #18 7:10 APIC version 21
(XEN) Processor #2 7:10 APIC version 21
(XEN) Processor #20 7:10 APIC version 21
(XEN) Processor #4 7:10 APIC version 21
(XEN) Processor #22 7:10 APIC version 21
(XEN) Processor #6 7:10 APIC version 21
(XEN) Processor #17 7:10 APIC version 21
(XEN) Processor #1 7:10 APIC version 21
(XEN) Processor #19 7:10 APIC version 21
(XEN) Processor #3 7:10 APIC version 21
(XEN) Processor #21 7:10 APIC version 21
(XEN) Processor #5 7:10 APIC version 21
(XEN) Processor #23 7:10 APIC version 21
(XEN) Processor #7 7:10 APIC version 21
(XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
(XEN) IOAPIC[1]: apic_id 1, version 32, address 0xfec80000, GSI 32-55
(XEN) Enabling APIC mode:  Phys.  Using 2 I/O APICs
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2793.111 MHz processor.
(XEN) Initing memory sharing.
(XEN) [VT-D]Disabling Interrupt remapping due to Intel 5500/5520/X58 Chipset errata #47, #53
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping not enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Passthrough
(XEN) Interrupt remapping disabled
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 64 KiB.
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB
(XEN) Brought up 16 CPUs
(XEN) Testing NMI watchdog --- CPU#0 okay. CPU#1 okay. CPU#2 okay. CPU#3 okay. CPU#4 okay. CPU#5 okay. CPU#6 okay. CPU#7 okay. CPU#8 okay. CPU#9 okay. CPU#10 okay. CPU#11 okay. CPU#12 okay. CPU#13 okay. CPU#14 okay. CPU#15 okay.
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 32-bit, PAE, lsb, paddr 0x100000 -> 0x62e000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   00000000cd000000->00000000ce000000 (186377 pages to be allocated)
(XEN)  Init. ramdisk: 000000122f809000->000000122ffff800
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: 00000000c0100000->00000000c062e000
(XEN)  Init. ramdisk: 00000000c062e000->00000000c0e24800
(XEN)  Phys-Mach map: 00000000c0e25000->00000000c0ee1000
(XEN)  Start info:    00000000c0ee1000->00000000c0ee14b4
(XEN)  Page tables:   00000000c0ee2000->00000000c0ef0000
(XEN)  Boot stack:    00000000c0ef0000->00000000c0ef1000
(XEN)  TOTAL:         00000000c0000000->00000000c1000000
(XEN)  ENTRY ADDRESS: 00000000c0100000
(XEN) Dom0 has maximum 4 VCPUs
(XEN) Scrubbing Free RAM: ..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 276kB init memory.
(XEN) __csched_vcpu_acct_start: setting dom 0 as the privileged domain
(XEN) Disabling C-states C3 and C6 on Nehalem Processors due to errata
``

CitrixNewbJD

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

While you are working on that, this is a good time to discuss triage. In all honesty, this is an old, unmaintained cluster. What is the hardware of the compute nodes?

The storage is poor. Even if we fix things, the system is very old in both software and hardware. So it's only a bandaid. Which is very doable, but it is what it is.

Doing a migration, like right now, might be the logical path forward. Let's get your specs for the current cluster, concerns for a new one and discuss if that's a real possibility. Also, what is the moment to moment impact of the outage?

I actually have a quote for a cluster from Scale that I had planned to try to push forward with the owner of the company when he returns from Europe next week. In the interest of properly giving you the information that you need, I want to be sure you're aware of that.

So the quote is already spec'd and ready to go? Maybe call him in Europe and see if things can be expedited? Would suck to have a major recovery operation happen just to hold off a discussion by one week.

I'd need to talk to someone from Scale to understand the process and guarantees of it working, or the headaches involved before making that call. Can you tell me with whom you spoke, or have them contact my reps, Blake Dickmeyer and Frank Seal?

CitrixNewbJD

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
If you do this...
cd ..
It will take you back to the main logs. What is the output of this...
ls | grep -i xen
[root@xen2 ~]# ls | grep -i xen
xen2pooldb
[root@xen2 ~]#
What does this return...
pwd
/root
that explains a lot - I think you want to be in /var/logs
 cd /var/logs
Yes
[root@xen2 log]# ls | grep -i xen
xen
xen-dmesg
xensource.log
xensource.log.1.gz
xensource.log.2.gz
xensource.log.3.gz
xensource.log.4.gz
xenstored-access.log
There we go. Let's see what the xen logs thing.
tail -n 100 xen

tail -n 100 xen-dmesg

tail -n 100 xensource.log

[root@xen2 log]# tail -n 100 /var/logs/xen
tail: cannot open `/var/logs/xen' for reading: No such file or directory
[root@xen2 log]# tail -n 100 /var/log/xen
tail: error reading `/var/log/xen': Is a directory
[root@xen2 log]# tail -n 100 xen
tail: error reading `xen': Is a directory
[root@xen2 log]# cd xen
[root@xen2 xen]# tail -n 100 xen
tail: cannot open `xen' for reading: No such file or directory

What's wrong with my syntax?

CitrixNewbJD

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
If you do this...
cd ..
It will take you back to the main logs. What is the output of this...
ls | grep -i xen
[root@xen2 ~]# ls | grep -i xen
xen2pooldb
[root@xen2 ~]#
What does this return...
pwd
/root
that explains a lot - I think you want to be in /var/logs
 cd /var/logs
Yes

[root@xen2 log]# ls | grep -i xen
xen
xen-dmesg
xensource.log
xensource.log.1.gz
xensource.log.2.gz
xensource.log.3.gz
xensource.log.4.gz
xenstored-access.log

CitrixNewbJD

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
If you do this...
cd ..
It will take you back to the main logs. What is the output of this...
ls | grep -i xen
[root@xen2 ~]# ls | grep -i xen
xen2pooldb
[root@xen2 ~]#
What does this return...
pwd

/root

CitrixNewbJD

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

While you are working on that, this is a good time to discuss triage. In all honesty, this is an old, unmaintained cluster. What is the hardware of the compute nodes?

The storage is poor. Even if we fix things, the system is very old in both software and hardware. So it's only a bandaid. Which is very doable, but it is what it is.

Doing a migration, like right now, might be the logical path forward. Let's get your specs for the current cluster, concerns for a new one and discuss if that's a real possibility. Also, what is the moment to moment impact of the outage?

I actually have a quote for a cluster from Scale that I had planned to try to push forward with the owner of the company when he returns from Europe next week. In the interest of properly giving you the information that you need, I want to be sure you're aware of that.

CitrixNewbJD

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

If you do this...
cd ..
It will take you back to the main logs. What is the output of this...
ls | grep -i xen

[root@xen2 ~]# ls | grep -i xen
xen2pooldb
[root@xen2 ~]#

CitrixNewbJD

0_1482866750909_Screenshot 2016-12-27 13.25.40.png

CitrixNewbJD

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@CitrixNewbJD

Sorry, ate my text again. Unless I should be looking somewhere else...

Get us the SMlog tail...
tail -n 100 SMlog

Dec 27 13:24:41 xen2 SM: [6456]   |- 5:0:0:1 sdf 8:80  failed faulty running
Dec 27 13:24:41 xen2 SM: [6456]   `- 7:0:0:1 sdl 8:176 active ready  running
Dec 27 13:24:41 xen2 SM: [6456] multipathd>
Dec 27 13:24:41 xen2 SM: [6456] MPATH:  Set val: [1, 4]
Dec 27 13:24:41 xen2 SM: [6456] Matched SCSIid, updating 360024e800070ed06000007f54dba7bfb
Dec 27 13:24:41 xen2 SM: [6456] MPATH: Updating entry for [360024e800070ed06000007f54dba7bfb], current: [0, 4]
Dec 27 13:24:41 xen2 SM: [6456] MPATH: Update done
Dec 27 13:24:41 xen2 SM: [6456] lock: closed /var/lock/sm/mpathcount2/host
Dec 27 13:24:41 xen2 SM: [6456] lock: released /var/lock/sm/mpathcount1/host
Dec 27 13:24:41 xen2 SM: [6456] lock: closed /var/lock/sm/mpathcount1/host
Dec 27 13:24:41 xen2 SM: [6579] lock: acquired /var/lock/sm/mpathcount1/host
Dec 27 13:24:41 xen2 SM: [6579] lock: released /var/lock/sm/mpathcount2/host
Dec 27 13:24:41 xen2 SM: [6579] MPATH: I get the lock
Dec 27 13:24:41 xen2 SM: [6579] Matched SCSIid, updating 360024e80007b786a000004134a9670f3
Dec 27 13:24:41 xen2 SM: [6579] MPATH: Updating entry for [360024e80007b786a000004134a9670f3], current: [2, 4]
Dec 27 13:24:41 xen2 SM: [6579] Matched SCSIid, updating 360024e800054baef0000ddd550c85082
Dec 27 13:24:41 xen2 SM: [6579] MPATH: Updating entry for [360024e800054baef0000ddd550c85082], current: [4, 4]
Dec 27 13:24:41 xen2 SM: [6579] Matched SCSIid, updating 360024e800070ed06000004314a967309
Dec 27 13:24:41 xen2 SM: [6579] MPATH: Updating entry for [360024e800070ed06000004314a967309], current: [1, 4]
Dec 27 13:24:41 xen2 SM: [6579] mpath cmd: show map 360024e800070ed06000004314a967309 topology
Dec 27 13:24:41 xen2 SM: [6663] MPATH: Trying to acquire the lock
Dec 27 13:24:41 xen2 SM: [6663] lock: tried lock /var/lock/sm/mpathcount1/host, acquired: False (exists: True)
Dec 27 13:24:41 xen2 SM: [6663] lock: tried lock /var/lock/sm/mpathcount2/host, acquired: True (exists: True)
Dec 27 13:24:41 xen2 SM: [6663] Failed to lock /var/lock/sm/mpathcount1/host on first attempt, blocked by PID 6579
Dec 27 13:24:41 xen2 SM: [6666] MPATH: Trying to acquire the lock
Dec 27 13:24:41 xen2 SM: [6666] lock: tried lock /var/lock/sm/mpathcount1/host, acquired: False (exists: True)
Dec 27 13:24:41 xen2 SM: [6666] lock: tried lock /var/lock/sm/mpathcount2/host, acquired: False (exists: True)
Dec 27 13:24:41 xen2 SM: [6666] lock: closed /var/lock/sm/mpathcount2/host
Dec 27 13:24:41 xen2 SM: [6666] lock: closed /var/lock/sm/mpathcount1/host
Dec 27 13:24:41 xen2 SM: [6579] mpath output: multipathd> 360024e800070ed06000004314a967309 dm-0 DELL,MD3000i
Dec 27 13:24:41 xen2 SM: [6579] size=1.4T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
Dec 27 13:24:41 xen2 SM: [6579] |-+- policy='round-robin 0' prio=4 status=active
Dec 27 13:24:41 xen2 SM: [6579] | |- 6:0:0:1 sdi 8:128 active ghost  running
Dec 27 13:24:41 xen2 SM: [6579] | `- 4:0:0:1 sdc 8:32  failed faulty running
Dec 27 13:24:41 xen2 SM: [6579] `-+- policy='round-robin 0' prio=3 status=enabled
Dec 27 13:24:41 xen2 SM: [6579]   |- 5:0:0:1 sdf 8:80  failed faulty running
Dec 27 13:24:41 xen2 SM: [6579]   `- 7:0:0:1 sdl 8:176 active ready  running
Dec 27 13:24:41 xen2 SM: [6579] multipathd>
Dec 27 13:24:41 xen2 SM: [6579] mpath cmd: show map 360024e800070ed06000004314a967309 topology
Dec 27 13:24:41 xen2 SM: [6579] mpath output: multipathd> 360024e800070ed06000004314a967309 dm-0 DELL,MD3000i
Dec 27 13:24:41 xen2 SM: [6579] size=1.4T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
Dec 27 13:24:41 xen2 SM: [6579] |-+- policy='round-robin 0' prio=4 status=active
Dec 27 13:24:41 xen2 SM: [6579] | |- 6:0:0:1 sdi 8:128 active ghost  running
Dec 27 13:24:41 xen2 SM: [6579] | `- 4:0:0:1 sdc 8:32  failed faulty running
Dec 27 13:24:41 xen2 SM: [6579] `-+- policy='round-robin 0' prio=3 status=enabled
Dec 27 13:24:41 xen2 SM: [6579]   |- 5:0:0:1 sdf 8:80  failed faulty running
Dec 27 13:24:41 xen2 SM: [6579]   `- 7:0:0:1 sdl 8:176 active ready  running
Dec 27 13:24:41 xen2 SM: [6579] multipathd>
Dec 27 13:24:41 xen2 SM: [6579] MPATH:  Set val: [2, 4]
Dec 27 13:24:41 xen2 SM: [6579] Matched SCSIid, updating 360024e800070ed06000007f54dba7bfb
Dec 27 13:24:41 xen2 SM: [6579] MPATH: Updating entry for [360024e800070ed06000007f54dba7bfb], current: [0, 4]
Dec 27 13:24:41 xen2 SM: [6579] MPATH: Update done
Dec 27 13:24:41 xen2 SM: [6579] lock: closed /var/lock/sm/mpathcount2/host
Dec 27 13:24:41 xen2 SM: [6579] lock: released /var/lock/sm/mpathcount1/host
Dec 27 13:24:41 xen2 SM: [6579] lock: closed /var/lock/sm/mpathcount1/host
Dec 27 13:24:41 xen2 SM: [6663] lock: acquired /var/lock/sm/mpathcount1/host
Dec 27 13:24:41 xen2 SM: [6663] lock: released /var/lock/sm/mpathcount2/host
Dec 27 13:24:41 xen2 SM: [6663] MPATH: I get the lock
Dec 27 13:24:41 xen2 SM: [6663] Matched SCSIid, updating 360024e80007b786a000004134a9670f3
Dec 27 13:24:41 xen2 SM: [6663] MPATH: Updating entry for [360024e80007b786a000004134a9670f3], current: [2, 4]
Dec 27 13:24:41 xen2 SM: [6663] Matched SCSIid, updating 360024e800054baef0000ddd550c85082
Dec 27 13:24:41 xen2 SM: [6663] MPATH: Updating entry for [360024e800054baef0000ddd550c85082], current: [4, 4]
Dec 27 13:24:41 xen2 SM: [6663] Matched SCSIid, updating 360024e800070ed06000004314a967309
Dec 27 13:24:41 xen2 SM: [6663] MPATH: Updating entry for [360024e800070ed06000004314a967309], current: [2, 4]
Dec 27 13:24:41 xen2 SM: [6663] mpath cmd: show map 360024e800070ed06000004314a967309 topology
Dec 27 13:24:41 xen2 SM: [6663] mpath output: multipathd> 360024e800070ed06000004314a967309 dm-0 DELL,MD3000i
Dec 27 13:24:41 xen2 SM: [6663] size=1.4T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
Dec 27 13:24:41 xen2 SM: [6663] |-+- policy='round-robin 0' prio=4 status=active
Dec 27 13:24:41 xen2 SM: [6663] | |- 6:0:0:1 sdi 8:128 active ghost  running
Dec 27 13:24:41 xen2 SM: [6663] | `- 4:0:0:1 sdc 8:32  failed faulty running
Dec 27 13:24:41 xen2 SM: [6663] `-+- policy='round-robin 0' prio=3 status=enabled
Dec 27 13:24:41 xen2 SM: [6663]   |- 5:0:0:1 sdf 8:80  failed faulty running
Dec 27 13:24:41 xen2 SM: [6663]   `- 7:0:0:1 sdl 8:176 active ready  running
Dec 27 13:24:41 xen2 SM: [6663] multipathd>
Dec 27 13:24:41 xen2 SM: [6663] mpath cmd: show map 360024e800070ed06000004314a967309 topology
Dec 27 13:24:42 xen2 SM: [6663] mpath output: multipathd> 360024e800070ed06000004314a967309 dm-0 DELL,MD3000i
Dec 27 13:24:42 xen2 SM: [6663] size=1.4T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
Dec 27 13:24:42 xen2 SM: [6663] |-+- policy='round-robin 0' prio=4 status=active
Dec 27 13:24:42 xen2 SM: [6663] | |- 6:0:0:1 sdi 8:128 active ghost  running
Dec 27 13:24:42 xen2 SM: [6663] | `- 4:0:0:1 sdc 8:32  failed faulty running
Dec 27 13:24:42 xen2 SM: [6663] `-+- policy='round-robin 0' prio=3 status=enabled
Dec 27 13:24:42 xen2 SM: [6663]   |- 5:0:0:1 sdf 8:80  failed faulty running
Dec 27 13:24:42 xen2 SM: [6663]   `- 7:0:0:1 sdl 8:176 active ready  running
Dec 27 13:24:42 xen2 SM: [6663] multipathd>
Dec 27 13:24:42 xen2 SM: [6663] Matched SCSIid, updating 360024e800070ed06000007f54dba7bfb
Dec 27 13:24:42 xen2 SM: [6663] MPATH: Updating entry for [360024e800070ed06000007f54dba7bfb], current: [0, 4]
Dec 27 13:24:42 xen2 SM: [6663] MPATH: Update done
Dec 27 13:24:42 xen2 SM: [6663] lock: closed /var/lock/sm/mpathcount2/host
Dec 27 13:24:42 xen2 SM: [6663] lock: released /var/lock/sm/mpathcount1/host
Dec 27 13:24:42 xen2 SM: [6663] lock: closed /var/lock/sm/mpathcount1/host
Dec 27 13:24:42 xen2 SM: [6728] MPATH: Trying to acquire the lock
Dec 27 13:24:42 xen2 SM: [6728] lock: tried lock /var/lock/sm/mpathcount1/host, acquired: True (exists: True)
Dec 27 13:24:42 xen2 SM: [6728] MPATH: I get the lock
Dec 27 13:24:42 xen2 SM: [6728] Matched SCSIid, updating 360024e80007b786a000004134a9670f3
Dec 27 13:24:42 xen2 SM: [6728] MPATH: Updating entry for [360024e80007b786a000004134a9670f3], current: [2, 4]
Dec 27 13:24:42 xen2 SM: [6728] Matched SCSIid, updating 360024e800054baef0000ddd550c85082
Dec 27 13:24:42 xen2 SM: [6728] MPATH: Updating entry for [360024e800054baef0000ddd550c85082], current: [4, 4]
Dec 27 13:24:42 xen2 SM: [6728] Matched SCSIid, updating 360024e800070ed06000004314a967309
Dec 27 13:24:42 xen2 SM: [6728] MPATH: Updating entry for [360024e800070ed06000004314a967309], current: [2, 4]
Dec 27 13:24:42 xen2 SM: [6728] mpath cmd: show map 360024e800070ed06000004314a967309 topology

CitrixNewbJD

Is there any way to tell why the hosts can't even attach to their own local storage, or why it was unplugged in the first place?

CitrixNewbJD

@CitrixNewbJD

Sorry, ate my text again. Unless I should be looking somewhere else...