I've inherited a XenServer 6.2 environment. My background is in Hyper-V. I understand some concepts, but I have not used these, or the Dell PowerVault SAN that it's attached to via iSCSI, so bear with me if I am lacking in background.

Current Status - XenPool with two hosts, Xen1 and Xen2 (Pool Master). VMs show as running, but can't interact with them in any way. All forms of storage are disconnected from the hosts. When I ran iscsiadmn node and session, from the console, Xen2 showed as being connected to the SAN. However, I cannot connect either host to their local storage. Both stuck in maintenance mode.

Background, and of course, I can flesh it out more, as questions arise. I'm going on 26 hours of work, -4 hours of sleep. Forgive any holes. I walked in yesterday to do maintenance when no one was here. I noticed I couldn't RDP into the servers that were managed by Xen1. I logged in, and not being familiar enough, I didn't notice if the storage connectivity was an issue right away. At the time, it seemed like there were networking issues. I couldn't force down any VMs, I couldn't move any VMs to Xen2. Tried to reboot the host, couldn't reboot it from XenCenter. Eventually had to force the server down. Over time, I saw that there was a with the control domain getting full, at 91%. Doing research, I cleared out some log files to get it under 3GB.

I repeatedly had issues with Xen1 going into maintenance mode after a reboot, and not being able to exit. Only resolved when I changed anything about its network settings. Eventually, I was commanded to restart both servers. After they came back up, Xen2 also lost all connection to its storage, and is now stuck in maintenance mode. Which means we have no servers at all, including Active Directory. I have internet by pointing DNS to my router.

Help!

@scottalanmiller Thank you. I've been an IT professional, or in IT leadership (but still hands-on) since the 90's. I took on a role a few months ago with a organization that's been patched all to heck and back, and I've been fighting crisis after crisis to try to make them fundamentally sound. I'm primarily a Windows/AD/Networking guy, and Scott brought me here as a resource for my latest crisis, XenServer.

I'm Dallas based, a husband and father, and I have a passion for worldwide travel. If you're looking to find a deal to get somewhere, I'm a good resource to have in your pocket.

On the phone with Dell, and the logs indicate that the two drives went down within seconds of each other, which may indicate that one of the drives isn't truly dead. They're going to try to revive it. Fingers crossed.

@scottalanmiller He's been talking to his most trusted employee, a developer. Apparently he doesn't even understand that that Xen and the SAN are two completely different things...

I do plan on updating my LinkedIn profile and resume over the next few days.

@NerdyDad Close. Friend of the family. He's also a really good guy, and probably a genius. He's on my side. Not necessarily a correlation to his genius, just sayin'.

@scottalanmiller

@scottalanmiller said in If you are new drop in say hello and introduce yourself please!:

@dominica and I are full time world travelers. Based in Dallas more than anywhere else in the US (own a home there) we've been living outside of the US for the last two years. Have been to 31 countries while living out of Spain, Panama, Nicaragua, Greece (Crete) and Romania. We leave for a week's vacation in Rome in 19 days and from there going to our new home in Noto, Siciliy.

I'll be in Venice for a couple of days mid-January, and taking the family to Rome for a week in mid-February.

This was my 2016 travel. http://www.gcmap.com/mapui?P=dfw-clt-jfk-auh%2C+dxb-doh-bkk-hkg-hnd-sfo-dfw dfw-msy-dfw dfw-sfo-dfw dfw-lax-dfw dfw-las-dfw dfw-las-dfw dfw-san-dfw dfw-lax-hnd-sfo-lax-dfw dfw-lax-syd-sin-icn-sfo-dfw dfw-lax-syd-dfw dfw-lax-jfk-mia-pty-mia-jfk-lax-aus-dfw dfw-las-dfw dfw-yvr-phx-san-mia-pty-mia-dfw dfw-dca-dfw dfw-aus-dfw dfw-jfk-zrh-lhr-dfw dfw-lhr-cdg-dfw tyr-dfw-nrt-hkg-dfw-tyr&MS=wls&DU=mi

@scottalanmiller I had gone to another user's machine to log into the PowerVault interface last night. I'm installing the software on my machine now.

Also, we're willing to engage someone formally to get this resolved.

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

I think that we need to look at the logs. Do you have access to get onto the machines themselves via SSH?

I haven't tried yet. Let me get a hold of putty.

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

I think that we need to look at the logs. Do you have access to get onto the machines themselves via SSH?

I'm logged into Xen2 with SSH now.

@scale It appears we may not be in as good as shape as we thought. The drives going into the SAN are failing. Trying to move everything off now.

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

That's more business consulting, but something I always say to people looking to start a business.... if you are the owner, what are you bringing to the table that keeps people from just running the business without you? Is it fame, marketing, sales, investment dollars, a truly brilliant idea, property, experience, hard work, special knowledge, etc. If you don't have something that makes you critical, your team will be better off being your competitor than your staff.

A work visa. He's British/Canadian.

@NerdyDad Close. Friend of the family. He's also a really good guy, and probably a genius. He's on my side. Not necessarily a correlation to his genius, just sayin'.

@scottalanmiller He's been talking to his most trusted employee, a developer. Apparently he doesn't even understand that that Xen and the SAN are two completely different things...

I do plan on updating my LinkedIn profile and resume over the next few days.

@travisdh1 @seal Full backup created with what? As I noted yesterday there is no backup solution in place as of yet, unless you're referring to something I'm not grokking.

Some VM OSes aren't coming up. Getting this on most (all?) of the VMs that had their OS on the downed SAN.

@Dashrender @scottalanmiller Systems are up. New disk in, VD is degraded and being rebuilt.

On the phone with Dell, and the logs indicate that the two drives went down within seconds of each other, which may indicate that one of the drives isn't truly dead. They're going to try to revive it. Fingers crossed.

@scottalanmiller @ghani Thanks for making me no longer the rookie!

@scottalanmiller Howabout TravelingJay? I assume usernames are not case-sensitive for login?