Posts made by jrc

So this weekend I updated my Xen hosts, and re-booted them both. So everything is 100% up to date. The reboot did not, however, fix my space issue. I am still using 2Tb more than I should. I also turned off Unitrends for the last few days. So I am 100% sure that I do not have any snapshots on there, at least none that show up in Xencenter.

Xencenter reports that I am using 6173.8Gb but only have 4115.7 allocated, 2 TB more used then there should be.

Now when I run xe vdi-list is-a-snapshot=true I get:

           uuid ( RO) : 5535a3db-da4f-4211-afa8-077241f63221
      name-label ( RW): Staff Home
name-description ( RW): VDI for staff home folders
         sr-uuid ( RO): 4558cecd-d90d-3259-7ea5-09478d0e386c
    virtual-size ( RO): 2,193,654,546,432
        sharable ( RO): false
       read-only ( RO): true

So I tried to delete this VDI with xe vdi-destroy uuid=5535a3db-da4f-4211-afa8-077241f63221, I get:

This operation cannot be performed because the system does not manage this VDI
vdi: 5535a3db-da4f-4211-afa8-077241f63221 (Staff Home)

A reclaim freed space does not make a difference (and only takes about 5 seconds to run).

So any suggestions on where I can go from here? Moving this VM to other storage and then back is not really an options, since I can't take this VM down for the time it would take to move all 2TB (hours and hours), since this is where all my user's home folders are.

@momurda said in Xenserver Space Woes:

Did you try using vgs and vhd-util, to see if the hidden=1 is on for your troubled vhd?
THere is also an XS6.5 update that was just released (well just showed up in my XC the other day), XS65ESP1029, with the description "fixes to Storage and Dom0 kernel modules"

I also just noticed your post yesterday in this thread about "Run out of space while coalescing".

I still think this is something to do with you Unitrends failures and snapshotting.
At this point i think all you can really do is reboot and/or delete those snapshots-but-not-snapshots, or just move the disk to another SR. However, befopre you do any of this, you should do something to save the data that could go missing.
I would robocopy /mir /copy:DATSO the shares on that virtual disk drive somewhere else just in case you wipe outthe virtual disk. I keep a 4TB usb3 drive hooked up to my desktop just for things like this.

Unfortunately I don't have a place to dump 2Tb of home folders, and even if I did, I don't have the luxury of taking everyone's folder offline for the amount of time that would take.

The VGS and VHD tool did yield an entire list of VHDs and showed me their relationships, but I can't seem to correlate the 2 VHDs to any of the vhd-util output, though they do appear to be gone.

I am thinking that Unitrends is definitely behind this odd issue. So I have turned off all the backup jobs for now, so once the job currently running (backup of this VM with the 2TB drive) is complete, I'll run all these commands again and see what it looks like. Plus I reboot and update all the Xen hosts before I turn them back on.

@momurda said in Xenserver Space Woes:

What does your storage server say about all this?

Nada. It's happily running as intended.

Well something changed overnight, becuase now when I look at the storage useage it says:

Used: 6505.4Gb and Allocated: 6734.8Gb

So why on earth has my allocated amount gone up by nearly 2587Gb?

sigh this is very very odd...

EDIT: Just did a rescan, and now it says Used:6505.4 Allocated: 4147.8. So it's back to "normal"

@momurda said in Xenserver Space Woes:

Ok, well then, I am just about out of ideas.
Is your Unitrends doing a backup now? At what point do your backups fail? Quickly? during bacup? During cleanup after backup(deleting snapshots, etc).
I also use Unitrends and had some problems at the beginning.
Perhaps i can help fix the Unitrends problem before you go about deleting 2TB vdis that you may or may not need.

The problem comes in when it goes to attach the snapshot to the backup appliance, it fails. And this not because of the maximum number of allowed VDIs on a VM, I ruled this out when I re-did the jobs so that no more than 8 VDIs are attached at any given time (plus the 4 backup targets and system drive, totalling 13, which is less than the 16 max that is allowed).

Not sure what it does after that, though given that I don't have a million orphaned snapshots I have to assume the it is deleting them after this failure. Something must've gone wrong with these 2 though.

Here is an example of a failed log (I get these for almost every backup attempt):

Files to backup:
-=-=-==-=-=-=-=
Client Name....: localhost.localdomain [127.0.0.1]
Starting Dir...: /
Command-line...: /usr/bp/bin/xprotect/xprotect --server 10.1.1.12 --username root --password ******** --backup --uuid <redacted>
Exit Code......: 0
Exit Status....: Failed
----- XProtect Messages ----
attaching the disk to vm failed
----- End XProtect Messages ----
No client messages due to log level 3
See client log in /usr/bp/logs.dir/bpserver-5.log
The database update failed. (Attempt 1 of 3)
The database update failed.
The command was "/usr/bp/bin/updatedb -b 1283 -f 1283 -a remove -l /tmp/file8JppsG > /dev/null 2>&1 ".

@momurda said in Xenserver Space Woes:

Edit: Have you selected View Hidden Objects from View menu?

Yes, still not there.

@momurda

I don't have a Virtual Disks option that I can find, certainly nothing that looks like what you have in the tree. How did you get there? How did you get it to show up like that?

NM, that's in the Object View. And no, the two disks I mentioned earlier do not show up in that list either.

@momurda said in Xenserver Space Woes:

Ok, what about the SR where you Unitrends backup disks are? Is there anything amiss there?

Nope. Shows only the backup target VDIs (4 of them, each 2044Gb)

@momurda said in Xenserver Space Woes:

If you goto the Storage in your Xencenter (i assume yorue using xencenter to look at things) Go to the SR with trouble, and look at the Storage tab for that SR. In the disks section you may see those 2 'orphaned' vdi, as well as the real one.
Is that the case? are they listed in Xencenter>SR>Storage?

The are not listed there. The only VDIs listed there are the ones actually being used by all my VMs.

@brianlittlejohn said in Xenserver Space Woes:

Make sure you have a backup before you reboot.

Well so you see there's the rub. I suspect my backups aren't working because of this issue, and the potential fix is to reboot. So I am damned if I do, and I am damned if I don't.

@dafyre said in Xenserver Space Woes:

@jrc said in Xenserver Space Woes:

@DustinB3403 said in Xenserver Space Woes:

~ Reboot now! No time for the weekend.. If it's gonna break its gonna break on company time damn it! ~

Well it's not broken now, everything is working as expected. So if I can schedule the down time it'll be better for everyone. However I am getting a "Run out of space while coalescing," which come to think on it may also be related...

Oh man... You just messed up. Now it is definitely going to break!

Well shit.... looks widely around for some wood to know on

@DustinB3403 said in Xenserver Space Woes:

~ Reboot now! No time for the weekend.. If it's gonna break its gonna break on company time damn it! ~

Well it's not broken now, everything is working as expected. So if I can schedule the down time it'll be better for everyone. However I am getting a "Run out of space while coalescing," which come to think on it may also be related...

@DustinB3403 said in Xenserver Space Woes:

@jrc

Are you planning to stick with Unitrends and just fix it?

Well at this point if Unitrends would start to work, then yes we'll stick with it, at least until the support contract we paid for is up, at which point we'll go back out and see what other options are available to us. If I can't get Unitrends to work without having to re-write the code myself, then we may end up ditching it.

@DustinB3403 said in Xenserver Space Woes:

If this is a snapshot issue, and you have no snapshots shown on the VM >Snapshots tab, for this VM I'd rescan your storage, and see if it is cleaned up.

Otherwise you'll have to reboot the system, let the system scan the storage.

There are no snapshots listed in the VM's snapshot tab, and when I rescan the storage, nothing seems to happen or at least nothing changes. And as I said, I'll be rebooting the f-ing things this weekend.

@DustinB3403 said in Xenserver Space Woes:

What kind of retention / backup system do you have in place?

A non-working Unitrends implementation (another freaking thing I am trying to solve, and am starting to think it may be related to this problem).

@scottalanmiller said in Xenserver Space Woes:

@dafyre said in Xenserver Space Woes:

It looks as if his XenServer is reporting that the VMs are using 2x the space that they should be.

Can we determine if it is exactly 200%? If so... maybe the SAN is doing some kind of internal replication?

I am nearly 100% sure the SAN is not doing anything more than presenting it's space to Xen. The fact that it's close to double the allocated is, I think, merely a co-incidence based on the fact that one of the VMs has a 2Tb VDI assigned to it that is mostly full. So I am pretty sure the problem is some sort of snapshot issue.

When I run xe vdi-list is-a-snapshot=true I get:

uuid ( RO) : 53d7329c-00fe-4659-a9fa-779e6341637e
name-label ( RW): Staff Home
name-description ( RW): VDI for staff home folders
sr-uuid ( RO): 4558cecd-d90d-3259-7ea5-09478d0e386c
virtual-size ( RO): 2193654546432
sharable ( RO): false
read-only ( RO): true

uuid ( RO) : 586f6e9a-bf7d-4fc4-89aa-6568ba91cea5
name-label ( RW): Staff Home
name-description ( RW): VDI for staff home folders
sr-uuid ( RO): 4558cecd-d90d-3259-7ea5-09478d0e386c
virtual-size ( RO): 2193654546432
sharable ( RO): false
read-only ( RO): true

Notice that both seem to be snapshots, and each is roughly 2Tb in size. So I think these are my culprits. Though what I can do with them I have no idea...

@scottalanmiller said in Xenserver Space Woes:

So the shared LUN is 11TB? Just to be sure that the SAN interface shows an 11TB LUN as being provided.

Yes. And yes, it is an IPOD in that I do have a single point of failure, though there is not too much I can do about it for now.

@DustinB3403 said in Xenserver Space Woes:

As I research this, have you restarted your systems to see if it clears out anything that might be hung there?

That is the plan this weekend. To totally shut down both hosts (after applying updates). Then to bring them back up. Really hoping that this fixes it.

@DustinB3403 said in Xenserver Space Woes:

Also on a side note, how many servers do you have here, and what kind of servers are these?

This sounds like an IPOD.

IPOD?

2 Xen Hosts (HP ML385p Gen 8, 96Gb of RAM, dual 12 core CPUs), shared SAN storage (HP 1040), directly connected via 8Gb/s fiber to each host and is multipathed to each as well. Total VMs is 16.

@DustinB3403 said in Xenserver Space Woes:

Did you configure your XenServers with Thin provisioning? How much space is consumed by your VM's, 4TB?

I often think to check for any snapshots that might be there, but you've already checked that.

It's been so long that I don't recall if I did thin provisioning or not, though I doubt it. but how do I check? As to how much space is consumed, also not sure, but I can tell you the max it could be is right around 4Tb (when I add up all the drive sizes I get 4139.5).

Xen Center reports 7378.5Gb Used, 4147.8Gb allocated out of 11166.3Gb

This may be a simple question, but can someone give me direction on how to resolve the fact that I have a shared SR for my 2 hosts (it's a SAN) that has 11Tb of useable space with just over 4Tb allocated, yet XenCenter says I am using almost 8Tb on it. There are no snapshots that I can find in Xencenter, so I am not sure what is eating up the extra 4Tb and I have no idea how to even go about finding it and clearing it. When I run the "reclaim freed space" it takes about 5 seconds to run and does nothing (at least nothing I can see).

So does anyone have some insight on how I can clean this up and release the 4Tb in question?

Xenserver 6.5 Update 8, with plans to install all the current updates this weekend.

Thanks in advance!