XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!

CitrixNewbJD

[root@xen2 log]# xe pbd-list
uuid ( RO)                  : 360fee57-907f-2c6a-e15d-166412f7bd7e
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): e58b6f8a-225d-2a83-642d-0610812efbea
         device-config (MRO): multihomelist: 192.168.130.101:3260,192.168.130.102:3260,192.168.130.103:3260,192.168.130.104:3260; SCSIid: 360024e800070ed06000004314a967309; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800070ed06000000004a61f4c6; port: 3260; target: 192.168.130.101
    currently-attached ( RO): false


uuid ( RO)                  : c4ee5bb1-f824-52c4-dc09-c2c50034801e
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): e8c1da6c-9016-fa1d-b357-e1bc070b4d2b
         device-config (MRO): multihomelist: 192.168.130.101:3260,192.168.130.102:3260,192.168.130.103:3260,192.168.130.104:3260; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800070ed06000000004a61f4c6; port: 3260; SCSIid: 360024e800070ed06000007f54dba7bfb; target: 192.168.130.101
    currently-attached ( RO): false


uuid ( RO)                  : 5dedd15d-45c6-c38d-7ed1-0eb8bd7a4620
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): d457d592-56c2-73e0-379a-26a55be27538
         device-config (MRO): iso_path: /XenISO; location: //192.168.1.230/Zipfiles; type: cifs; username: spindleinc\mgould; cifspassword_secret: c74bebaa-bbb0-e2d8-9a19-e11433375cd4
    currently-attached ( RO): false


uuid ( RO)                  : 5bd81e5b-895e-d385-bd28-da4a4b2603f8
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): 169601bb-0d24-006a-a02f-e6ccb65a560f
         device-config (MRO): device: /dev/sdb4
    currently-attached ( RO): false


uuid ( RO)                  : ebf92343-b5b1-d530-ebd8-235365c3d961
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): ff6303ba-0852-6f9c-b59f-ddb2cbc4d07b
         device-config (MRO): multihomelist: 192.168.130.108:3260,192.168.130.105:3260,192.168.130.107:3260,192.168.130.106:3260; SCSIid: 360024e800054baef0000ddd550c85082; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800054baef0000000050c83cdb; port: 3260; target: 192.168.130.108
    currently-attached ( RO): false


uuid ( RO)                  : 0f241fac-bf50-5141-69aa-7c31ac0fb406
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): ec1432db-9f89-2a04-987e-263f0d30c66f
         device-config (MRO): iso_path: /Software/Microsoft/Windows/Server/2012; location: //192.168.1.100/files; type: cifs; username: spindleinc\mgould; cifspassword_secret: 77701771-8d4e-3de7-a562-90542842828a
    currently-attached ( RO): false


uuid ( RO)                  : 6fd14ddd-9595-88fe-592f-1d289d06ebbc
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): 7e001a4b-4f2d-15fe-4e7a-653062b4cdcd
         device-config (MRO): location: /dev/xapi/cd
    currently-attached ( RO): false


uuid ( RO)                  : e97a8dbc-eec8-dcde-63ef-fe0f350ed73d
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): 4e97fda1-40cb-8a4a-baba-184a9e81fee5
         device-config (MRO): location: /opt/xensource/packages/iso; legacy_mode: true
    currently-attached ( RO): false


uuid ( RO)                  : 7ab79e27-cbe1-04b0-4c14-f8e89c200411
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): 4e97fda1-40cb-8a4a-baba-184a9e81fee5
         device-config (MRO): location: /opt/xensource/packages/iso; legacy_mode: true
    currently-attached ( RO): false


uuid ( RO)                  : cacb5cd4-3894-8fbc-e3ce-6b342c11df95
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): e8c1da6c-9016-fa1d-b357-e1bc070b4d2b
         device-config (MRO): multihomelist: 192.168.130.101:3260,192.168.130.102:3260,192.168.130.103:3260,192.168.130.104:3260; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800070ed06000000004a61f4c6; port: 3260; SCSIid: 360024e800070ed06000007f54dba7bfb; target: 192.168.130.101
    currently-attached ( RO): false


uuid ( RO)                  : 09b657e0-974e-7a9e-99f8-425e8da32c7b
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): dbdb8c1c-30b6-5e24-991d-7f42c452fd07
         device-config (MRO): multihomelist: 192.168.130.101:3260,192.168.130.102:3260,192.168.130.103:3260,192.168.130.104:3260; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800070ed06000000004a61f4c6; port: 3260; SCSIid: 360024e80007b786a000004134a9670f3; target: 192.168.130.101
    currently-attached ( RO): false


uuid ( RO)                  : 38ac4cf3-15b4-b319-2fc7-44eff01b93f6
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): 49447156-88bc-e5cf-7578-c0a5775d6dc7
         device-config (MRO): device: /dev/disk/by-id/scsi-360024e8072e05b0011fe4fc707bd94eb-part4
    currently-attached ( RO): false


uuid ( RO)                  : 8ffccb24-5c1e-27a9-879d-01e89b419fca
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): 85d27d5d-19cd-df91-62e2-f64cfb7153cf
         device-config (MRO): location: /dev/xapi/block
    currently-attached ( RO): false


uuid ( RO)                  : 6d49b0e1-5d6a-9779-1f23-5c4414121518
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): 5cdb03b4-746d-4ee1-8a45-3b3a33895e69
         device-config (MRO): type: cifs; location: //192.168.1.230/zipfiles; cifspassword_secret: 25876439-7bf0-75be-f977-17ce22ec3536; username: jason; iso_path: /microsoft
    currently-attached ( RO): false


uuid ( RO)                  : 24695844-f828-d80e-a002-08107734c527
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): e58b6f8a-225d-2a83-642d-0610812efbea
         device-config (MRO): multihomelist: 192.168.130.101:3260,192.168.130.102:3260,192.168.130.103:3260,192.168.130.104:3260; SCSIid: 360024e800070ed06000004314a967309; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800070ed06000000004a61f4c6; port: 3260; target: 192.168.130.101
    currently-attached ( RO): false


uuid ( RO)                  : 80f89268-eb01-5460-5ed9-e65b8fe62875
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): ff6303ba-0852-6f9c-b59f-ddb2cbc4d07b
         device-config (MRO): multihomelist: 192.168.130.108:3260,192.168.130.107:3260,192.168.130.105:3260,192.168.130.106:3260; SCSIid: 360024e800054baef0000ddd550c85082; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800054baef0000000050c83cdb; port: 3260; target: 192.168.130.108
    currently-attached ( RO): false


uuid ( RO)                  : 965ead53-f00f-158a-db6d-84865dcdb38d
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): d457d592-56c2-73e0-379a-26a55be27538
         device-config (MRO): iso_path: /XenISO; location: //192.168.1.230/Zipfiles; type: cifs; username: spindleinc\mgould; cifspassword_secret: a44f6bfd-dd64-768d-56e6-cb1c002c5d8c
    currently-attached ( RO): false


uuid ( RO)                  : dbef4921-c3f1-2cb8-6548-d462745bb1ea
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): 5cdb03b4-746d-4ee1-8a45-3b3a33895e69
         device-config (MRO): type: cifs; location: //192.168.1.230/zipfiles; cifspassword_secret: d7287cdb-b3a8-8583-aba4-a2c24d0394d5; username: jason; iso_path: /microsoft
    currently-attached ( RO): false


uuid ( RO)                  : 9b968975-b85e-5369-86f3-d924c9d10871
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): dbdb8c1c-30b6-5e24-991d-7f42c452fd07
         device-config (MRO): multihomelist: 192.168.130.101:3260,192.168.130.102:3260,192.168.130.103:3260,192.168.130.104:3260; targetIQN: iqn.1984-05.com.dell:powervault.md3000i.60024e800070ed06000000004a61f4c6; port: 3260; SCSIid: 360024e80007b786a000004134a9670f3; target: 192.168.130.101
    currently-attached ( RO): false


uuid ( RO)                  : c15826f2-0f0b-6fcf-87ab-36b90754368f
             host-uuid ( RO): 3e2674e3-b204-471d-a451-d641545b0892
               sr-uuid ( RO): ba8349a9-87d5-19b0-36b6-c12f4e5b40a9
         device-config (MRO): location: /dev/xapi/cd
    currently-attached ( RO): false


uuid ( RO)                  : 65de2fdc-49dd-c0a4-b397-c44e42e6b998
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): b03b52b1-e403-1649-9595-8f7c1b442896
         device-config (MRO): location: /dev/xapi/block
    currently-attached ( RO): false


uuid ( RO)                  : b602d816-38fc-d5f8-6880-943a3ba824d3
             host-uuid ( RO): c89ae6e4-1649-4a1c-9180-6c7d2385ac18
               sr-uuid ( RO): ec1432db-9f89-2a04-987e-263f0d30c66f
         device-config (MRO): iso_path: /Software/Microsoft/Windows/Server/2012; location: //192.168.1.100/files; type: cifs; username: spindleinc\mgould; cifspassword_secret: 8ca82529-cbac-4818-db7e-ccdcbb85c193
    currently-attached ( RO): false

momurda

Who is spindleinc\mgould? Is your Xenserver setup with AD integration? it looks like this is the case. Is mgould a disabled account in AD(this seems to be the guy who had your job, maybe). If so, that is a huge issue.
Is there a reason you have 5 ISO libraries?

CitrixNewbJD

@momurda said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

Who is spindleinc\mgould? Is your Xenserver setup with AD integration? it looks like this is the case. Is mgould a disabled account in AD(this seems to be the guy who had your job, maybe). If so, that is a huge issue.
Is there a reason you have 5 ISO libraries?

mgould is still with us. I don't know if it was set up with AD integration, but I would assume so based on this. It's not a disabled account, but we can't get to AD.

I have no idea why we have 5 ISO libraries.

momurda

Oh, you cant authenticate to ad but all your SR need it to attach
That is the problem right there i think.

scottalanmiller

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@momurda said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

Who is spindleinc\mgould? Is your Xenserver setup with AD integration? it looks like this is the case. Is mgould a disabled account in AD(this seems to be the guy who had your job, maybe). If so, that is a huge issue.
Is there a reason you have 5 ISO libraries?

mgould is still with us. I don't know if it was set up with AD integration, but I would assume so based on this. It's not a disabled account, but we can't get to AD.

That's not good. AD was built on top of a system that was merged with all shared dependencies... so AD needs Xen which needs the PowerVault, which needs Xen which needs AD. So the circular issue potentially? There is no AD to fail over to if anything goes wrong with the Xen cluster has no redundancy. No matter how many copies run on XenServer here, there is only one disk array holding it all.

scottalanmiller

@momurda said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

Oh, you cant authenticate to ad but all your SR need it to attach
That is the problem right there i think.

That would do it. Excellent catch.

CitrixNewbJD

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@momurda said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

Who is spindleinc\mgould? Is your Xenserver setup with AD integration? it looks like this is the case. Is mgould a disabled account in AD(this seems to be the guy who had your job, maybe). If so, that is a huge issue.
Is there a reason you have 5 ISO libraries?

mgould is still with us. I don't know if it was set up with AD integration, but I would assume so based on this. It's not a disabled account, but we can't get to AD.

That's not good. AD was built on top of a system that was merged with all shared dependencies... so AD needs Xen which needs the PowerVault, which needs Xen which needs AD. So the circular issue potentially? There is no AD to fail over to if anything goes wrong with the Xen cluster has no redundancy. No matter how many copies run on XenServer here, there is only one disk array holding it all.

Does this mean it's irreparable?

scottalanmiller

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@momurda said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

Who is spindleinc\mgould? Is your Xenserver setup with AD integration? it looks like this is the case. Is mgould a disabled account in AD(this seems to be the guy who had your job, maybe). If so, that is a huge issue.
Is there a reason you have 5 ISO libraries?

mgould is still with us. I don't know if it was set up with AD integration, but I would assume so based on this. It's not a disabled account, but we can't get to AD.

That's not good. AD was built on top of a system that was merged with all shared dependencies... so AD needs Xen which needs the PowerVault, which needs Xen which needs AD. So the circular issue potentially? There is no AD to fail over to if anything goes wrong with the Xen cluster has no redundancy. No matter how many copies run on XenServer here, there is only one disk array holding it all.

Does this mean it's irreparable?

No, just really REALLY silly.

scottalanmiller

@momurda from a quick glance those appear to be all CIFS ISO repos that are AD tied, so we should be able to just remove those to fix that, right? I don't see anything else.

momurda

@CitrixNewbJD
No i dont think so.
You got some work to do though
Probably need to do:
log into the san and change the creds for iscsi, disable ad authentication in xc, and then probably forget/reattach the sr

NerdyDad

I always try to make sure that I have an administrative local user to every system, just in case I cannot get to AD.

scottalanmiller

@momurda said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@CitrixNewbJD
No i dont think so.
You got some work to do though
Probably need to do:
log into the san and change the creds for iscsi, disable ad authentication in xc, and then probably forget/reattach the sr

I didn't see it on the iSCSI, darn it. Still fixable, but more work.

scottalanmiller

@NerdyDad said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

I always try to make sure that I have an administrative local user to every system, just in case I cannot get to AD.

That, and never have AD on top of something that depends on it. That's like locking your keys in your car.

NerdyDad

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@NerdyDad said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

I always try to make sure that I have an administrative local user to every system, just in case I cannot get to AD.

That, and never have AD on top of something that depends on it. That's like locking your keys in your car.

Yup. Our SAN is isolated from the rest of the house network physically. All authentication between the SAN and the host should never even touch AD just in case of things like this.

Sometimes playing what-if is a good exercise to keep bad things from happening as far as security, reliability, and recovery.

momurda

@scottalanmiller
Yes the cifs repos can be forgotten for now. I doubt there is anything important there in the iso repo.
@CitrixNewbJD
Before you do anything else, make sure you have the root pw for your xs servers as disabling AD integration will force you to use them.

momurda

Unless you have a DC that you can get working, ie one not attached to your shared storage. Perhaps you have a physical dc somewhere? Unlikely i know.

seal

@CitrixNewbJD This is Frank. I hope Scott and the gang here can help get you back operational quickly. To touch on a topic that was mentioned earlier, VMs from Xen can be imported and converted on the fly. As long as the Scale nodes can browse to the VM files (.vhd) on the storage, the XenServer functional state doesn't matter.

CitrixNewbJD

@momurda

Having been through this once before, and learning the hard way, I do normally have a physical DC. Despite my warnings, because I know that we do not currently have one here, I was told to bring it all down. And here we are. We do not have a physical DC.

CitrixNewbJD

@momurda

I've been using the root authentication for everything.

CitrixNewbJD

@seal

Frank, can we speak on the phone for a minute so I can be sure I can intelligently talk to the guy when demanding his money?