Xenserver Space Woes
-
@jrc said in Xenserver Space Woes:
@DustinB3403 said in Xenserver Space Woes:
~ Reboot now! No time for the weekend.. If it's gonna break its gonna break on company time damn it! ~
Well it's not broken now, everything is working as expected. So if I can schedule the down time it'll be better for everyone. However I am getting a "Run out of space while coalescing," which come to think on it may also be related...
Oh man... You just messed up. Now it is definitely going to break!
-
@dafyre said in Xenserver Space Woes:
@jrc said in Xenserver Space Woes:
@DustinB3403 said in Xenserver Space Woes:
~ Reboot now! No time for the weekend.. If it's gonna break its gonna break on company time damn it! ~
Well it's not broken now, everything is working as expected. So if I can schedule the down time it'll be better for everyone. However I am getting a "Run out of space while coalescing," which come to think on it may also be related...
Oh man... You just messed up. Now it is definitely going to break!
Well shit.... looks widely around for some wood to know on
-
Make sure you have a backup before you reboot.
-
@brianlittlejohn said in Xenserver Space Woes:
Make sure you have a backup before you reboot.
Well so you see there's the rub. I suspect my backups aren't working because of this issue, and the potential fix is to reboot. So I am damned if I do, and I am damned if I don't.
-
If you goto the Storage in your Xencenter (i assume yorue using xencenter to look at things) Go to the SR with trouble, and look at the Storage tab for that SR. In the disks section you may see those 2 'orphaned' vdi, as well as the real one.
Is that the case? are they listed in Xencenter>SR>Storage? -
@momurda said in Xenserver Space Woes:
If you goto the Storage in your Xencenter (i assume yorue using xencenter to look at things) Go to the SR with trouble, and look at the Storage tab for that SR. In the disks section you may see those 2 'orphaned' vdi, as well as the real one.
Is that the case? are they listed in Xencenter>SR>Storage?The are not listed there. The only VDIs listed there are the ones actually being used by all my VMs.
-
Ok, what about the SR where you Unitrends backup disks are? Is there anything amiss there?
-
@momurda said in Xenserver Space Woes:
Ok, what about the SR where you Unitrends backup disks are? Is there anything amiss there?
Nope. Shows only the backup target VDIs (4 of them, each 2044Gb)
-
Hmmm. I ran xe vdi-list is-a-snapshot=true
Got 1 result, a disk from our crm server. If you go to
the Objects button instead of Infrastructure in XC(
You see in mine here, the 2 disk for crm01. One of them shows as a snapshot, even though the XC thinks there isnt one for this vm. This is a byproduct of failed backups i think with Unitrends, soemtimes they dont delete themselves.
If you do this you may see your 2 vdi listed here as well as the disk actually being used.
Ah weird formatting on the pic, soz. -
I don't have a Virtual Disks option that I can find, certainly nothing that looks like what you have in the tree. How did you get there? How did you get it to show up like that?
NM, that's in the Object View. And no, the two disks I mentioned earlier do not show up in that list either.
-
This is the Objects view, not Infrastructure view in Xencenter.
From there expand Virtual Disks, and you will see this view.
Edit: Have you selected View Hidden Objects from View menu?
The problem seems to be these 2 massive disks that are there taking up space, but not being seen by XC correctly. I am certain you could manually delete them, but i would make sure the 'real' disk that you are currently using has a different uuid than the 2 that are in limbo. -
@momurda said in Xenserver Space Woes:
Edit: Have you selected View Hidden Objects from View menu?
Yes, still not there.
-
Ok, well then, I am just about out of ideas.
Is your Unitrends doing a backup now? At what point do your backups fail? Quickly? during bacup? During cleanup after backup(deleting snapshots, etc).
I also use Unitrends and had some problems at the beginning.
Perhaps i can help fix the Unitrends problem before you go about deleting 2TB vdis that you may or may not need. -
@momurda said in Xenserver Space Woes:
Ok, well then, I am just about out of ideas.
Is your Unitrends doing a backup now? At what point do your backups fail? Quickly? during bacup? During cleanup after backup(deleting snapshots, etc).
I also use Unitrends and had some problems at the beginning.
Perhaps i can help fix the Unitrends problem before you go about deleting 2TB vdis that you may or may not need.The problem comes in when it goes to attach the snapshot to the backup appliance, it fails. And this not because of the maximum number of allowed VDIs on a VM, I ruled this out when I re-did the jobs so that no more than 8 VDIs are attached at any given time (plus the 4 backup targets and system drive, totalling 13, which is less than the 16 max that is allowed).
Not sure what it does after that, though given that I don't have a million orphaned snapshots I have to assume the it is deleting them after this failure. Something must've gone wrong with these 2 though.
Here is an example of a failed log (I get these for almost every backup attempt):
Files to backup:
-=-=-==-=-=-=-=
Client Name....: localhost.localdomain [127.0.0.1]
Starting Dir...: /
Command-line...: /usr/bp/bin/xprotect/xprotect --server 10.1.1.12 --username root --password ******** --backup --uuid <redacted>
Exit Code......: 0
Exit Status....: Failed
----- XProtect Messages ----
attaching the disk to vm failed
----- End XProtect Messages ----
No client messages due to log level 3
See client log in /usr/bp/logs.dir/bpserver-5.log
The database update failed. (Attempt 1 of 3)
The database update failed.
The command was "/usr/bp/bin/updatedb -b 1283 -f 1283 -a remove -l /tmp/file8JppsG > /dev/null 2>&1 ". -
I was looking for more ways to look at virtual storage in Xenserver
http://devsops.blogspot.com/2013/02/how-to-use-vhd-util-tool-in-xenserver.html
This post may be useful, uses vgs and vhd-util. This could give you a bit of insight into the storage -
Also with regards to your backups, you may have to look at bpserver-5.log to actually see the error in the backup. The attach the disk to vm failed seems to me a generic error for Unitrends. Ive always foudn more useful info int he bpserver log file.
-
http://discussions.citrix.com/topic/324186-too-many-base-copy-when-xe-vdi-list/
More fun with vhd-util.
this guy at citrix forums says moving the disk to new SR, then runnign Reclaim Free Space may do something. -
Well something changed overnight, becuase now when I look at the storage useage it says:
Used: 6505.4Gb and Allocated: 6734.8Gb
So why on earth has my allocated amount gone up by nearly 2587Gb?
sigh this is very very odd...
EDIT: Just did a rescan, and now it says Used:6505.4 Allocated: 4147.8. So it's back to "normal"
-
What does your storage server say about all this?
-
@momurda said in Xenserver Space Woes:
What does your storage server say about all this?
Nada. It's happily running as intended.
