Today is the day from Hell!



  • For starters, my phone system stopped making outgoing calls starting at 10:45 AM - didn't get that fixed until 6:15 PM.

    Then my XO box won't allow me to logon, I know I have the right password, yet it refuses to let me logon, after several more things I hard reboot it and it won't boot.

    So now I'm in scramble mode to try to see if the SD card is bad or it's full.

    Anyone willing to offer assistance is greatly appreciated.



  • I'm currently creating a Mint Live USB stick.

    My plan is to boot a laptop off it and then look at the SD card to see if it's full - if so, find logs and kill them, then try to boot the server from it.

    If not - then I guess I get to reinstall XS on a new SD card - and pry that I can import my SR and recover my VMs.



  • Sorry 😞



  • I purchased two new 32 GB SD cards and am now booting from the XS install media.

    Here are the instructions Bill found

    http://support.citrix.com/article/CTX136342



  • So I popped in a new SD Card and installed XS 6.5 again. Went to do updates and decide to just start at the oldest and move my way upward. Well once I reached Sp1 suddenly there was about 35 more - no thanks, don't want to be here forever, already late enough.

    Found this link: http://support.citrix.com/article/CTX138115
    Updated June 14, 2016

    It shows that you can go Install > Sp1 > update 12 > update 24 > update 27 > update 29 > update 32 > update 33.

    Many of the updates are cumulative, do you don't have to install them all from scratch.

    Hope this saves someone some time.



  • OK another snag. Bill's link never mentioned the possibility that this isn't an LVM partition (which it's not, it's ext) so my re-introduce command is wrong.

    ug - I wonder if my SR is destroyed?

    *edit

    I feel that it's important to get as much information to anyone reading this thread for troubleshooting reasons as early in the thread as possible as such, enjoy the following.

    Step number 11 you are introducing the old storage repository to your system. The current instructions do not have you confirm the type of partition that you have.
    During a typical install of XS 6.5 you are given two options (through a checkbox) to either full provision VMs or Thin Provision them. The default is a non checked box which means your VMs will be full provisioned and your SR will be lvm. But if you're like me, I checked the box because i wanted Thin Provisioning, my partition was ext.

    This is a long winded way to say, check your partition type first. If you have an ext partition type, change the type=lvm to type=ext in your xe sr-introduce command.

    Also in step 14, the last line, which is a UUID, is bolded and appears to be part of the command you're suppose to execute. But really, this is the output from the command. You'll use this output in step 15.



  • This, I think, is beautiful!

    https://i.imgur.com/mdAmIbC.png

    I believe that this means that my SR still has all of my disks on it. Now to create the VMs and assign the disks to them.



  • Next major piece of success - Got XO running again. Now to build a Windows 2012 R2 VM and assign the VDIs to it.



  • OK, the best site I've seen, well, so far today.

    https://i.imgur.com/gSN9vAJ.png

    All of my servers are back up and running.



  • @Dashrender said in Today is the day from Hell!:

    OK another snag. Bill's link never mentioned the possibility that this isn't an LVM partition (which it's not, it's ext) so my re-introduce command is wrong.

    ug - I wonder if my SR is destroyed?

    This bit missing from the Citrix documentation cost me at least 1 hour, but probably more like 2.



  • This thread was helpful in removing the SR and PBD so I could reattach them correctly.

    http://support.citrix.com/article/CTX131328



  • and major props to

    @Stefan Lissmats

    For providing this excellent post

    Quite old thread but I wanted to contribute with what I strongly believe is the answer.
    I ran into the exact same issue today and also tried the renaming solution you tried.
    But the solution is quit simple, or it was for me anyway.
    The prefix XSLocalEXT is not a lvm volume but i ext based volume.
    So in the end what i simply did was to run the sr-introduce with the type ext instead of lvm
    So your command would be sr-introduce uuid=43d6cf1c-3b86-dd4f-8afc-9dcf2688082c type=ext >name-label="Local storage" content-type=user

    Over here

    Here was my error

    Command

    xe pbd-plug uuid=9dba453d-db22-57e5-0504-55d2d1dbb6d5
    

    Error code: SR_BACKEND_FAILURE_47
    Error parameters: , The SR is not available [opterr=no such volume group: VG_XenStorage-3c68d5fc-2e9f-5f5b-b832-dc1aacf920b4],

    Comparing the above output including VG_XenStorage which did not match my XSLocalEXT let me to that post.



  • @Dashrender said in Today is the day from Hell!:

    @Dashrender said in Today is the day from Hell!:

    OK another snag. Bill's link never mentioned the possibility that this isn't an LVM partition (which it's not, it's ext) so my re-introduce command is wrong.

    ug - I wonder if my SR is destroyed?

    This bit missing from the Citrix documentation cost me at least 1 hour, but probably more like 2.

    Hope things are working again?

    About the documentation: Contribute back, tell them about the missing part.



  • Just FYI, this is what I use for patching. I have it setup on a cron schedule.

    https://github.com/antonym/xs_patcher



  • @thwr said in Today is the day from Hell!:

    @Dashrender said in Today is the day from Hell!:

    @Dashrender said in Today is the day from Hell!:

    OK another snag. Bill's link never mentioned the possibility that this isn't an LVM partition (which it's not, it's ext) so my re-introduce command is wrong.

    ug - I wonder if my SR is destroyed?

    This bit missing from the Citrix documentation cost me at least 1 hour, but probably more like 2.

    Hope things are working again?

    About the documentation: Contribute back, tell them about the missing part.

    Done.



  • Glad that things are back working. Sorry that I was only on telegram for most of this.



  • @Dashrender said

    Many of the updates are cumulative, do you don't have to install them all from scratch.

    That was I was discussing in my thread from a while back, that there is a specific order to do them

    The conclusion of that thread was that XO was smart enough to do it and should always be used, but if you don't have XO running, what you found is the best alternative I think.



  • I was a little confused about how exactly to recover from a XS boot failure. Obviously the way you did it was able to get the VMs back up and running. That is good to know.

    There is also a methodology to backup the entire XS system, including the VM metadata. This process is described here...
    http://techblog.danielpellarini.com/sysadmin/steps-to-take-to-restore-xenserver-from-backup/

    In another linked article, he explains what is backup up in the metadata as:
    "And by VM metadata I mean what drives are connected to what VM, the name of the drive, the network interfaces etc."

    I guess if you had a lot of custom settings for your VMs (tags, etc.) or a lot of stuff listed as metadata, it might be good to do such a backup.

    This is something I am still a bit unclear on.



  • @BRRABill said in Today is the day from Hell!:

    I guess if you had a lot of custom settings for your VMs (tags, etc.) or a lot of stuff listed as metadata, it might be good to do such a backup.

    This is something I am still a bit unclear on.

    Or just have like 20+ VMs (hell I'd want that at more than 5 for sure). You don't want to be guessing what VDI (I just feel icky typing that) - I mean virtual disks - go with what VM. Plus, as you said, what adapters are there, etc.



  • @coliver said in Today is the day from Hell!:

    Just FYI, this is what I use for patching. I have it setup on a cron schedule.

    https://github.com/antonym/xs_patcher

    Thanks!



  • @Dashrender said in Today is the day from Hell!:

    So I popped in a new SD Card and installed XS 6.5 again. Went to do updates and decide to just start at the oldest and move my way upward. Well once I reached Sp1 suddenly there was about 35 more - no thanks, don't want to be here forever, already late enough.

    Found this link: http://support.citrix.com/article/CTX138115
    Updated June 14, 2016

    It shows that you can go Install > Sp1 > update 12 > update 24 > update 27 > update 29 > update 32 > update 33.

    Many of the updates are cumulative, do you don't have to install them all from scratch.

    Hope this saves someone some time.

    This is what i did when upgraded past sp1, just installed about 3 or 4 cumulative updates.
    Also, in the future you could save a backup of your XS install with vm/sr info to the local SR, or anywhere else, just in case. It is an option from #xsconsole
    I do this before upgrading usually, and keep it there afterwards.
    edit: ah wait, i see someone mentioned the xsconsole backups.



  • This is the second time in two days I've read about XS console - what is the XS console? Do you mean, does Citrix mean, the XS host? Calling it a console is so..... weird.

    But hey, they call the virtual disks that are attached to VMs VDIs - so what do I know?



  • The root console
    0_1467304225843_upload-cd40beef-d1be-40fd-9bcf-65910c321bda
    The xsconsole
    0_1467304309417_upload-5c653bd6-d231-4d41-8065-344c4f3e2a5d



  • @Dashrender said

    This is the second time in two days I've read about XS console - what is the XS console? Do you mean, does Citrix mean, the XS host? Calling it a console is so..... weird.

    But hey, they call the virtual disks that are attached to VMs VDIs - so what do I know?

    PIPE DOWN! Do you want your servers hearing you and crashing again?????



  • @momurda said

    The xsconsole

    Was redacting that address really necessary? 🙂



  • probably not



  • @momurda said

    probably not

    If someone can find you with that, you've got a real hacker on your hands!



  • @momurda said in Today is the day from Hell!:

    The root console
    0_1467304225843_upload-cd40beef-d1be-40fd-9bcf-65910c321bda
    The xsconsole
    0_1467304309417_upload-5c653bd6-d231-4d41-8065-344c4f3e2a5d

    Psst - give greenshot a try - it's a free screen capture utility that has cool bluring features for bluring whatever you don't want read.



  • @Dashrender Hope your having a better day today 🙂



  • @aaronstuder said in Today is the day from Hell!:

    @Dashrender Hope your having a better day today 🙂

    I'm having an OK one. Now if we could just keep employees from quitting.


Log in to reply