ML
    • Recent
    • Categories
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Xenserver Space Woes

    IT Discussion
    9
    65
    13.4k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • DustinB3403D
      DustinB3403 @jrc
      last edited by

      @jrc said in Xenserver Space Woes:

      @scottalanmiller said in Xenserver Space Woes:

      @dafyre said in Xenserver Space Woes:

      It looks as if his XenServer is reporting that the VMs are using 2x the space that they should be.

      Can we determine if it is exactly 200%? If so... maybe the SAN is doing some kind of internal replication?

      I am nearly 100% sure the SAN is not doing anything more than presenting it's space to Xen. The fact that it's close to double the allocated is, I think, merely a co-incidence based on the fact that one of the VMs has a 2Tb VDI assigned to it that is mostly full. So I am pretty sure the problem is some sort of snapshot issue.

      When I run xe vdi-list is-a-snapshot=true I get:

      uuid ( RO) : 53d7329c-00fe-4659-a9fa-779e6341637e
      name-label ( RW): Staff Home
      name-description ( RW): VDI for staff home folders
      sr-uuid ( RO): 4558cecd-d90d-3259-7ea5-09478d0e386c
      virtual-size ( RO): 2193654546432
      sharable ( RO): false
      read-only ( RO): true

      uuid ( RO) : 586f6e9a-bf7d-4fc4-89aa-6568ba91cea5
      name-label ( RW): Staff Home
      name-description ( RW): VDI for staff home folders
      sr-uuid ( RO): 4558cecd-d90d-3259-7ea5-09478d0e386c
      virtual-size ( RO): 2193654546432
      sharable ( RO): false
      read-only ( RO): true

      Notice that both seem to be snapshots, and each is roughly 2Tb in size. So I think these are my culprits. Though what I can do with them I have no idea...

      If this is a snapshot issue, and you have no snapshots shown on the VM >Snapshots tab, for this VM I'd rescan your storage, and see if it is cleaned up.

      Otherwise you'll have to reboot the system, let the system scan the storage.

      1 Reply Last reply Reply Quote 0
      • DustinB3403D
        DustinB3403 @jrc
        last edited by

        @jrc

        Are you planning to stick with Unitrends and just fix it?

        jrcJ 1 Reply Last reply Reply Quote 0
        • jrcJ
          jrc @DustinB3403
          last edited by

          @DustinB3403 said in Xenserver Space Woes:

          @jrc

          Are you planning to stick with Unitrends and just fix it?

          Well at this point if Unitrends would start to work, then yes we'll stick with it, at least until the support contract we paid for is up, at which point we'll go back out and see what other options are available to us. If I can't get Unitrends to work without having to re-write the code myself, then we may end up ditching it.

          @DustinB3403 said in Xenserver Space Woes:

          If this is a snapshot issue, and you have no snapshots shown on the VM >Snapshots tab, for this VM I'd rescan your storage, and see if it is cleaned up.

          Otherwise you'll have to reboot the system, let the system scan the storage.

          There are no snapshots listed in the VM's snapshot tab, and when I rescan the storage, nothing seems to happen or at least nothing changes. And as I said, I'll be rebooting the f-ing things this weekend.

          DustinB3403D 1 Reply Last reply Reply Quote 0
          • DustinB3403D
            DustinB3403 @jrc
            last edited by

            @jrc said in Xenserver Space Woes:

            @DustinB3403 said in Xenserver Space Woes:

            @jrc

            Are you planning to stick with Unitrends and just fix it?

            Well at this point if Unitrends would start to work, then yes we'll stick with it, at least until the support contract we paid for is up, at which point we'll go back out and see what other options are available to us. If I can't get Unitrends to work without having to re-write the code myself, then we may end up ditching it.

            @DustinB3403 said in Xenserver Space Woes:

            If this is a snapshot issue, and you have no snapshots shown on the VM >Snapshots tab, for this VM I'd rescan your storage, and see if it is cleaned up.

            Otherwise you'll have to reboot the system, let the system scan the storage.

            There are no snapshots listed in the VM's snapshot tab, and when I rescan the storage, nothing seems to happen or at least nothing changes. And as I said, I'll be rebooting the f-ing things this weekend.

            ~ Reboot now! No time for the weekend.. If it's gonna break its gonna break on company time damn it! ~

            jrcJ 1 Reply Last reply Reply Quote 0
            • jrcJ
              jrc @DustinB3403
              last edited by

              @DustinB3403 said in Xenserver Space Woes:

              ~ Reboot now! No time for the weekend.. If it's gonna break its gonna break on company time damn it! ~

              Well it's not broken now, everything is working as expected. So if I can schedule the down time it'll be better for everyone. However I am getting a "Run out of space while coalescing," which come to think on it may also be related...

              dafyreD 1 Reply Last reply Reply Quote 1
              • dafyreD
                dafyre @jrc
                last edited by

                @jrc said in Xenserver Space Woes:

                @DustinB3403 said in Xenserver Space Woes:

                ~ Reboot now! No time for the weekend.. If it's gonna break its gonna break on company time damn it! ~

                Well it's not broken now, everything is working as expected. So if I can schedule the down time it'll be better for everyone. However I am getting a "Run out of space while coalescing," which come to think on it may also be related...

                Oh man... You just messed up. Now it is definitely going to break!

                jrcJ 1 Reply Last reply Reply Quote 0
                • jrcJ
                  jrc @dafyre
                  last edited by

                  @dafyre said in Xenserver Space Woes:

                  @jrc said in Xenserver Space Woes:

                  @DustinB3403 said in Xenserver Space Woes:

                  ~ Reboot now! No time for the weekend.. If it's gonna break its gonna break on company time damn it! ~

                  Well it's not broken now, everything is working as expected. So if I can schedule the down time it'll be better for everyone. However I am getting a "Run out of space while coalescing," which come to think on it may also be related...

                  Oh man... You just messed up. Now it is definitely going to break!

                  Well shit.... looks widely around for some wood to know on

                  1 Reply Last reply Reply Quote 0
                  • brianlittlejohnB
                    brianlittlejohn
                    last edited by

                    Make sure you have a backup before you reboot.

                    jrcJ 1 Reply Last reply Reply Quote 0
                    • jrcJ
                      jrc @brianlittlejohn
                      last edited by

                      @brianlittlejohn said in Xenserver Space Woes:

                      Make sure you have a backup before you reboot.

                      Well so you see there's the rub. I suspect my backups aren't working because of this issue, and the potential fix is to reboot. So I am damned if I do, and I am damned if I don't.

                      1 Reply Last reply Reply Quote 0
                      • momurdaM
                        momurda
                        last edited by

                        If you goto the Storage in your Xencenter (i assume yorue using xencenter to look at things) Go to the SR with trouble, and look at the Storage tab for that SR. In the disks section you may see those 2 'orphaned' vdi, as well as the real one.
                        Is that the case? are they listed in Xencenter>SR>Storage?

                        jrcJ 1 Reply Last reply Reply Quote 2
                        • jrcJ
                          jrc @momurda
                          last edited by

                          @momurda said in Xenserver Space Woes:

                          If you goto the Storage in your Xencenter (i assume yorue using xencenter to look at things) Go to the SR with trouble, and look at the Storage tab for that SR. In the disks section you may see those 2 'orphaned' vdi, as well as the real one.
                          Is that the case? are they listed in Xencenter>SR>Storage?

                          The are not listed there. The only VDIs listed there are the ones actually being used by all my VMs.

                          1 Reply Last reply Reply Quote 0
                          • momurdaM
                            momurda
                            last edited by

                            Ok, what about the SR where you Unitrends backup disks are? Is there anything amiss there?

                            jrcJ 1 Reply Last reply Reply Quote 0
                            • jrcJ
                              jrc @momurda
                              last edited by

                              @momurda said in Xenserver Space Woes:

                              Ok, what about the SR where you Unitrends backup disks are? Is there anything amiss there?

                              Nope. Shows only the backup target VDIs (4 of them, each 2044Gb)

                              1 Reply Last reply Reply Quote 0
                              • momurdaM
                                momurda
                                last edited by momurda

                                Hmmm. I ran xe vdi-list is-a-snapshot=true
                                Got 1 result, a disk from our crm server. If you go to
                                the Objects button instead of Infrastructure in XC(0_1466016707185_Capture.PNG
                                You see in mine here, the 2 disk for crm01. One of them shows as a snapshot, even though the XC thinks there isnt one for this vm. This is a byproduct of failed backups i think with Unitrends, soemtimes they dont delete themselves.
                                If you do this you may see your 2 vdi listed here as well as the disk actually being used.
                                Ah weird formatting on the pic, soz.

                                jrcJ 1 Reply Last reply Reply Quote 0
                                • jrcJ
                                  jrc @momurda
                                  last edited by jrc

                                  @momurda

                                  I don't have a Virtual Disks option that I can find, certainly nothing that looks like what you have in the tree. How did you get there? How did you get it to show up like that?

                                  NM, that's in the Object View. And no, the two disks I mentioned earlier do not show up in that list either.

                                  1 Reply Last reply Reply Quote 0
                                  • momurdaM
                                    momurda
                                    last edited by momurda

                                    This is the Objects view, not Infrastructure view in Xencenter.
                                    0_1466017105987_upload-a1689ce9-086c-46c4-b4cf-8cfcc2e42aeb
                                    From there expand Virtual Disks, and you will see this view.
                                    Edit: Have you selected View Hidden Objects from View menu?
                                    The problem seems to be these 2 massive disks that are there taking up space, but not being seen by XC correctly. I am certain you could manually delete them, but i would make sure the 'real' disk that you are currently using has a different uuid than the 2 that are in limbo.

                                    jrcJ 1 Reply Last reply Reply Quote 0
                                    • jrcJ
                                      jrc @momurda
                                      last edited by

                                      @momurda said in Xenserver Space Woes:

                                      Edit: Have you selected View Hidden Objects from View menu?

                                      Yes, still not there.

                                      1 Reply Last reply Reply Quote 0
                                      • momurdaM
                                        momurda
                                        last edited by

                                        Ok, well then, I am just about out of ideas.
                                        Is your Unitrends doing a backup now? At what point do your backups fail? Quickly? during bacup? During cleanup after backup(deleting snapshots, etc).
                                        I also use Unitrends and had some problems at the beginning.
                                        Perhaps i can help fix the Unitrends problem before you go about deleting 2TB vdis that you may or may not need.

                                        jrcJ 1 Reply Last reply Reply Quote 1
                                        • jrcJ
                                          jrc @momurda
                                          last edited by jrc

                                          @momurda said in Xenserver Space Woes:

                                          Ok, well then, I am just about out of ideas.
                                          Is your Unitrends doing a backup now? At what point do your backups fail? Quickly? during bacup? During cleanup after backup(deleting snapshots, etc).
                                          I also use Unitrends and had some problems at the beginning.
                                          Perhaps i can help fix the Unitrends problem before you go about deleting 2TB vdis that you may or may not need.

                                          The problem comes in when it goes to attach the snapshot to the backup appliance, it fails. And this not because of the maximum number of allowed VDIs on a VM, I ruled this out when I re-did the jobs so that no more than 8 VDIs are attached at any given time (plus the 4 backup targets and system drive, totalling 13, which is less than the 16 max that is allowed).

                                          Not sure what it does after that, though given that I don't have a million orphaned snapshots I have to assume the it is deleting them after this failure. Something must've gone wrong with these 2 though.

                                          Here is an example of a failed log (I get these for almost every backup attempt):

                                          Files to backup:
                                          -=-=-==-=-=-=-=
                                          Client Name....: localhost.localdomain [127.0.0.1]
                                          Starting Dir...: /
                                          Command-line...: /usr/bp/bin/xprotect/xprotect --server 10.1.1.12 --username root --password ******** --backup --uuid <redacted>
                                          Exit Code......: 0
                                          Exit Status....: Failed
                                          ----- XProtect Messages ----
                                          attaching the disk to vm failed
                                          ----- End XProtect Messages ----
                                          No client messages due to log level 3
                                          See client log in /usr/bp/logs.dir/bpserver-5.log
                                          The database update failed. (Attempt 1 of 3)
                                          The database update failed.
                                          The command was "/usr/bp/bin/updatedb -b 1283 -f 1283 -a remove -l /tmp/file8JppsG > /dev/null 2>&1 ".

                                          1 Reply Last reply Reply Quote 0
                                          • momurdaM
                                            momurda
                                            last edited by

                                            I was looking for more ways to look at virtual storage in Xenserver
                                            http://devsops.blogspot.com/2013/02/how-to-use-vhd-util-tool-in-xenserver.html
                                            This post may be useful, uses vgs and vhd-util. This could give you a bit of insight into the storage

                                            jrcJ 1 Reply Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 3
                                            • 4
                                            • 2 / 4
                                            • First post
                                              Last post