Posts made by anthonyh

@dustinb3403 said in XenServer 6.5 - Clean Up Storage Repository:

Doh. . .

This is a feature of XO directly. . . ha yea nevermind.

Well, I guess that means I should set it up.

@dustinb3403 said in XenServer 6.5 - Clean Up Storage Repository:

What output do you get from XS cli when you run this?

xapi-explore-sr

Hm. I don't seem to have that command. I have the following:

[root@vmhost10 ~]# xapi
xapi                     xapi-autostart-vms       xapi-db-process          xapi-wait-init-complete  
[root@vmhost10 ~]#

@dustinb3403 said in XenServer 6.5 - Clean Up Storage Repository:

@anthonyh said in XenServer 6.5 - Clean Up Storage Repository:

@dustinb3403 said in XenServer 6.5 - Clean Up Storage Repository:

@anthonyh said in XenServer 6.5 - Clean Up Storage Repository:

@dustinb3403 said in XenServer 6.5 - Clean Up Storage Repository:

@anthonyh said in XenServer 6.5 - Clean Up Storage Repository:

Last night I set up a new SR for Zimbra and live migrated the OS disk over to the new SR without any issues. Took ~5 minutes for the 20 GB VHD.

I'd like to do the same for the 1 TB VHD, but it makes me nervous...

If the process was to bomb mid-progress, what would happen? Is it easy to recover from?

You would have a backup, but why go through this process? Why not let the system run a coalesce? Is this a production system or your lab?

Production system. The problem is that there isn't enough space on the SR to perform a coalesce. I'm trying to avoid bringing Zimbra down if at all possible. Might not be possible, but I can at least try.

The coalesce process is likely already running attempting to clean up an old snapshot. Performing a manual SR scan, should clean up this issue.

Already done this several times. No dice.

I should probably just re-read the thread, but what type of backup are you running with this guest? You aren't using XO so I'm curious where/what is causing this issue.

I am using Alike's "enhanced backup" model, which is snapshot based backups. I also take snapshots of the VM (all VMs, really) whenever I do any sort of maintenance, so I can't really point the blame at Alike. I don't know how long the orphaned snapshots have been around. The interesting thing is that except for Saturday night's run (when I got the alert from the pool that the coalesce failed due to not enough disk space), backups have been successful. Backups aren't performed on Sundays.

I don't have the orphaned snapshots issue with any other VM that I'm aware of. SR usage everywhere else looks to be what is expected.

@dustinb3403 said in XenServer 6.5 - Clean Up Storage Repository:

@anthonyh said in XenServer 6.5 - Clean Up Storage Repository:

@dustinb3403 said in XenServer 6.5 - Clean Up Storage Repository:

@anthonyh said in XenServer 6.5 - Clean Up Storage Repository:

Last night I set up a new SR for Zimbra and live migrated the OS disk over to the new SR without any issues. Took ~5 minutes for the 20 GB VHD.

I'd like to do the same for the 1 TB VHD, but it makes me nervous...

If the process was to bomb mid-progress, what would happen? Is it easy to recover from?

You would have a backup, but why go through this process? Why not let the system run a coalesce? Is this a production system or your lab?

Production system. The problem is that there isn't enough space on the SR to perform a coalesce. I'm trying to avoid bringing Zimbra down if at all possible. Might not be possible, but I can at least try.

The coalesce process is likely already running attempting to clean up an old snapshot. Performing a manual SR scan, should clean up this issue.

Already done this several times. No dice.

@dustinb3403 said in XenServer 6.5 - Clean Up Storage Repository:

@anthonyh said in XenServer 6.5 - Clean Up Storage Repository:

@momurda said in XenServer 6.5 - Clean Up Storage Repository:

@anthonyh I have had a few disk migrations fail over the last 2 years. Most of the time you just end up with a broken vdi on the destination and the source is fine without issue.
But i have had to restart guest after disk migration failure before.

I'm OK with that. I will have to think about this...

This can mean unplanned downtime. Which for an exchange system can be costly. Granted the downtime is usually nominal but is worth considering allowing the system to clean up this issue at the nearest possible planned down time.

Yes, it's a gamble for sure.

@dustinb3403 said in XenServer 6.5 - Clean Up Storage Repository:

@anthonyh said in XenServer 6.5 - Clean Up Storage Repository:

Last night I set up a new SR for Zimbra and live migrated the OS disk over to the new SR without any issues. Took ~5 minutes for the 20 GB VHD.

I'd like to do the same for the 1 TB VHD, but it makes me nervous...

If the process was to bomb mid-progress, what would happen? Is it easy to recover from?

You would have a backup, but why go through this process? Why not let the system run a coalesce? Is this a production system or your lab?

Production system. The problem is that there isn't enough space on the SR to perform a coalesce. I'm trying to avoid bringing Zimbra down if at all possible. Might not be possible, but I can at least try.

@momurda said in XenServer 6.5 - Clean Up Storage Repository:

@anthonyh I have had a few disk migrations fail over the last 2 years. Most of the time you just end up with a broken vdi on the destination and the source is fine without issue.
But i have had to restart guest after disk migration failure before.

I'm OK with that. I will have to think about this...

Last night I set up a new SR for Zimbra and live migrated the OS disk over to the new SR without any issues. Took ~5 minutes for the 20 GB VHD.

I'd like to do the same for the 1 TB VHD, but it makes me nervous...

If the process was to bomb mid-progress, what would happen? Is it easy to recover from?

@dustinb3403 said in XenServer 6.5 - Clean Up Storage Repository:

The coalesce job in my experience usually doesn't take too long, but it's purely based on your host Sr performance.

And while you can kick it off manually once offline, you would still have no status.

Do you have XO? I believe it'll show you unhealthy SRs.

I do not have XenOrchestra. I should set that up...

@momurda said in XenServer 6.5 - Clean Up Storage Repository:

@anthonyh No shrinking of SRs.
Are there other vm on the SR you could move?

Nope, the SR is dedicated to Zimbra. I suppose I could create a new 2TB SR for Zimbra and move the disks, then delete the old SR when it's done. Though moving a 1TB VHD of this importance makes me nervous, haha.

Well...

I know XenServer supports growing SRs, but does it support shrinking SRs?

If so, I can grow the SR to 4 TB, perform the online coalesce, then shrink the SR back to 2 TB.

Thoughts?

From what I'm reading here, I either need to grow the SR or do an offline coalesce: https://support.citrix.com/article/CTX201296

SR_BACKEND_FAILURE_44 insufficient space:

The process of taking snapshots requires additional overhead on your SR. So you need sufficient room to perform the operation. For a running VM with a single snapshot to get coalesce you need twice the space in case of LVM SR’s (Active VDI + Single Snapshotting VDI). If we are in short of space in the SR, we get the following error.

Either do an offline coalesce or increase the SR size to accommodate online coalescing.

I could grow the SR, but I don't want to throw disk space at it for the sake of more disk space. Perhaps I'll need to plan some downtime to do an offline coalesce.

I suspect those hidden guys are what I want to get rid of...

Hmm, I'm not sure how to interpret the output...

[root@vmhost10 ~]# vhd-util scan -f -m "VHD-*" -l VG_XenStorage-8b007a92-c815-6d9a-d539-125db0c4b016 -p                                
vhd=VHD-c52a7680-b3fa-4ffd-8e73-a472067eb710 capacity=1099511627776 size=92312436736 hidden=1 parent=none                              
   vhd=VHD-00c565b0-ab40-4e6d-886e-41c51f62992a capacity=1099511627776 size=1100304351232 hidden=1 parent=VHD-c52a7680-b3fa-4ffd-8e73-a472067eb710
      vhd=VHD-1ab9f17f-eefa-4e62-9455-3e73623d1453 capacity=1099511627776 size=29364322304 hidden=1 parent=VHD-00c565b0-ab40-4e6d-886e-41c51f62992a
         vhd=VHD-586e7cc3-3fbc-4aa1-89bc-6974454aee7d capacity=1099511627776 size=1101667500032 hidden=0 parent=VHD-1ab9f17f-eefa-4e62-9455-3e73623d1453
vhd=VHD-1be106fb-8d7a-438a-98df-23438fc87f75 capacity=21474836480 size=21525168128 hidden=0 parent=none                                
[root@vmhost10 ~]#

Taking a look at this now: https://support.citrix.com/article/CTX139224

@momurda said in XenServer 6.5 - Clean Up Storage Repository:

Do you have Xencenter installed on a desktop?
If you choose View>Hidden Objects then look at the Storage Tab of the SR you might see the problem.

If not there is some cli stuff involving vhd-util

I do, and View > Hidden Objects is enabled. I only see the two disks in the list. For what it's worth I am using XenServer 7.2 since we have some 7.2 hosts in our environment (working on bringing every host up to 7.2). Not sure if this makes a difference.

@momurda As I said in the OP, none.

Yeah, I still have a XenServer 6.5 pool. Planning to upgrade to 7.2 in the near future, but until then...

Last night I got an alert from this XenServer pool about an SR having "No space left on device" due to "Run out of space while coalescing." Turns out the SR in question is one I have set up specifically for our Zimbra instance.

It's got two virtual disks: a 20 GB disk for the OS, and a 1 TB disk for the Zimbra install.

There are no snapshots on the VM. However, the SR claims to be just about full.

I've tried running the Storage > Reclaim Free Space in XenCenter, but that results in no change.

As a result, I cannot perform any snapshot based backups of the VM.

Any ideas on what I can try to get this cleaned up?

Ha, ok, I found a workaround. I can simply cat the message back into the mail spool.

cat $msgFile >> $mailSpool

Boom, message is back in the mail spool and my process re-consumes it.

@scottalanmiller said in Linux (CentOS) - fetchmail and mail spool:

I can't remember the lat time that I used fetchmail.

Well, ultimately, all I need are the messages dumped to file (one file per email). The way I'm achieving this now is by using fetchmail to dump the messages to the user's mail spool, then using formail to separate the messages into individual text files.

cat $mailSpool | formail -ds sh -c "cat > $stagingDir/msg."'$FILENO'

So, if there is a better way to approach this, perhaps in one step, I'm totally game.