Xenserver Space Woes
-
@DustinB3403 said in Xenserver Space Woes:
Also on a side note, how many servers do you have here, and what kind of servers are these?
This sounds like an IPOD.
IPOD?
2 Xen Hosts (HP ML385p Gen 8, 96Gb of RAM, dual 12 core CPUs), shared SAN storage (HP 1040), directly connected via 8Gb/s fiber to each host and is multipathed to each as well. Total VMs is 16.
-
@DustinB3403 said in Xenserver Space Woes:
As I research this, have you restarted your systems to see if it clears out anything that might be hung there?
That is the plan this weekend. To totally shut down both hosts (after applying updates). Then to bring them back up. Really hoping that this fixes it.
-
@jrc said in Xenserver Space Woes:
@DustinB3403 said in Xenserver Space Woes:
Also on a side note, how many servers do you have here, and what kind of servers are these?
This sounds like an IPOD.
IPOD?
2 Xen Hosts (HP ML385p Gen 8, 96Gb of RAM, dual 12 core CPUs), shared SAN storage (HP 1040), directly connected via 8Gb/s fiber to each host and is multipathed to each as well. Total VMs is 16.
That is exactly a description of an IPOD (Inverted Pyramid of Doom)
-
So the shared LUN is 11TB? Just to be sure that the SAN interface shows an 11TB LUN as being provided.
-
@jrc said in Xenserver Space Woes:
@DustinB3403 said in Xenserver Space Woes:
Did you configure your XenServers with Thin provisioning? How much space is consumed by your VM's, 4TB?
I often think to check for any snapshots that might be there, but you've already checked that.
It's been so long that I don't recall if I did thin provisioning or not, though I doubt it. but how do I check? As to how much space is consumed, also not sure, but I can tell you the max it could be is right around 4Tb (when I add up all the drive sizes I get 4139.5).
Xen Center reports 7378.5Gb Used, 4147.8Gb allocated out of 11166.3Gb
It does....
By the way... for more info on what an IPOD is... check out this thread... https://www.mangolassi.it/topic/8821/risk-3-2-1-stock-inverted-pyramid-design
-
@scottalanmiller said in Xenserver Space Woes:
So the shared LUN is 11TB? Just to be sure that the SAN interface shows an 11TB LUN as being provided.
Yes. And yes, it is an IPOD in that I do have a single point of failure, though there is not too much I can do about it for now.
-
It looks as if his XenServer is reporting that the VMs are using 2x the space that they should be.
-
@jrc said in Xenserver Space Woes:
@scottalanmiller said in Xenserver Space Woes:
So the shared LUN is 11TB? Just to be sure that the SAN interface shows an 11TB LUN as being provided.
Yes. And yes, it is an IPOD in that I do have a single point of failure, though there is not too much I can do about it for now.
It would make a good discussion for future planning... but on another thread. We'll try to stick to the capacity visibility point here. But well worth talking about separately.
-
@dafyre said in Xenserver Space Woes:
It looks as if his XenServer is reporting that the VMs are using 2x the space that they should be.
Which to me makes me think that there were snap's that didn't get cleaned up properly. A reboot generally resolves this kind of issue.
-
@dafyre said in Xenserver Space Woes:
It looks as if his XenServer is reporting that the VMs are using 2x the space that they should be.
Can we determine if it is exactly 200%? If so... maybe the SAN is doing some kind of internal replication?
-
@scottalanmiller said in Xenserver Space Woes:
@dafyre said in Xenserver Space Woes:
It looks as if his XenServer is reporting that the VMs are using 2x the space that they should be.
Can we determine if it is exactly 200%? If so... maybe the SAN is doing some kind of internal replication?
@jrc says... Xen Center reports 7378.5Gb Used, 4147.8Gb allocated out of 11166.3Gb
So, not quite 2x. But right close.
-
@dafyre said in Xenserver Space Woes:
@scottalanmiller said in Xenserver Space Woes:
@dafyre said in Xenserver Space Woes:
It looks as if his XenServer is reporting that the VMs are using 2x the space that they should be.
Can we determine if it is exactly 200%? If so... maybe the SAN is doing some kind of internal replication?
@jrc says... Xen Center reports 7378.5Gb Used, 4147.8Gb allocated out of 11166.3Gb
So, not quite 2x. But right close.
That's almost 300GB less than double the allocated. It's not the SAN doing something.
What kind of retention / backup system do you have in place?
-
@scottalanmiller said in Xenserver Space Woes:
@dafyre said in Xenserver Space Woes:
It looks as if his XenServer is reporting that the VMs are using 2x the space that they should be.
Can we determine if it is exactly 200%? If so... maybe the SAN is doing some kind of internal replication?
I am nearly 100% sure the SAN is not doing anything more than presenting it's space to Xen. The fact that it's close to double the allocated is, I think, merely a co-incidence based on the fact that one of the VMs has a 2Tb VDI assigned to it that is mostly full. So I am pretty sure the problem is some sort of snapshot issue.
When I run xe vdi-list is-a-snapshot=true I get:
uuid ( RO) : 53d7329c-00fe-4659-a9fa-779e6341637e
name-label ( RW): Staff Home
name-description ( RW): VDI for staff home folders
sr-uuid ( RO): 4558cecd-d90d-3259-7ea5-09478d0e386c
virtual-size ( RO): 2193654546432
sharable ( RO): false
read-only ( RO): trueuuid ( RO) : 586f6e9a-bf7d-4fc4-89aa-6568ba91cea5
name-label ( RW): Staff Home
name-description ( RW): VDI for staff home folders
sr-uuid ( RO): 4558cecd-d90d-3259-7ea5-09478d0e386c
virtual-size ( RO): 2193654546432
sharable ( RO): false
read-only ( RO): trueNotice that both seem to be snapshots, and each is roughly 2Tb in size. So I think these are my culprits. Though what I can do with them I have no idea...
-
@DustinB3403 said in Xenserver Space Woes:
What kind of retention / backup system do you have in place?
A non-working Unitrends implementation (another freaking thing I am trying to solve, and am starting to think it may be related to this problem).
-
@jrc said in Xenserver Space Woes:
@scottalanmiller said in Xenserver Space Woes:
@dafyre said in Xenserver Space Woes:
It looks as if his XenServer is reporting that the VMs are using 2x the space that they should be.
Can we determine if it is exactly 200%? If so... maybe the SAN is doing some kind of internal replication?
I am nearly 100% sure the SAN is not doing anything more than presenting it's space to Xen. The fact that it's close to double the allocated is, I think, merely a co-incidence based on the fact that one of the VMs has a 2Tb VDI assigned to it that is mostly full. So I am pretty sure the problem is some sort of snapshot issue.
When I run xe vdi-list is-a-snapshot=true I get:
uuid ( RO) : 53d7329c-00fe-4659-a9fa-779e6341637e
name-label ( RW): Staff Home
name-description ( RW): VDI for staff home folders
sr-uuid ( RO): 4558cecd-d90d-3259-7ea5-09478d0e386c
virtual-size ( RO): 2193654546432
sharable ( RO): false
read-only ( RO): trueuuid ( RO) : 586f6e9a-bf7d-4fc4-89aa-6568ba91cea5
name-label ( RW): Staff Home
name-description ( RW): VDI for staff home folders
sr-uuid ( RO): 4558cecd-d90d-3259-7ea5-09478d0e386c
virtual-size ( RO): 2193654546432
sharable ( RO): false
read-only ( RO): trueNotice that both seem to be snapshots, and each is roughly 2Tb in size. So I think these are my culprits. Though what I can do with them I have no idea...
If this is a snapshot issue, and you have no snapshots shown on the VM >Snapshots tab, for this VM I'd rescan your storage, and see if it is cleaned up.
Otherwise you'll have to reboot the system, let the system scan the storage.
-
Are you planning to stick with Unitrends and just fix it?
-
@DustinB3403 said in Xenserver Space Woes:
Are you planning to stick with Unitrends and just fix it?
Well at this point if Unitrends would start to work, then yes we'll stick with it, at least until the support contract we paid for is up, at which point we'll go back out and see what other options are available to us. If I can't get Unitrends to work without having to re-write the code myself, then we may end up ditching it.
@DustinB3403 said in Xenserver Space Woes:
If this is a snapshot issue, and you have no snapshots shown on the VM >Snapshots tab, for this VM I'd rescan your storage, and see if it is cleaned up.
Otherwise you'll have to reboot the system, let the system scan the storage.
There are no snapshots listed in the VM's snapshot tab, and when I rescan the storage, nothing seems to happen or at least nothing changes. And as I said, I'll be rebooting the f-ing things this weekend.
-
@jrc said in Xenserver Space Woes:
@DustinB3403 said in Xenserver Space Woes:
Are you planning to stick with Unitrends and just fix it?
Well at this point if Unitrends would start to work, then yes we'll stick with it, at least until the support contract we paid for is up, at which point we'll go back out and see what other options are available to us. If I can't get Unitrends to work without having to re-write the code myself, then we may end up ditching it.
@DustinB3403 said in Xenserver Space Woes:
If this is a snapshot issue, and you have no snapshots shown on the VM >Snapshots tab, for this VM I'd rescan your storage, and see if it is cleaned up.
Otherwise you'll have to reboot the system, let the system scan the storage.
There are no snapshots listed in the VM's snapshot tab, and when I rescan the storage, nothing seems to happen or at least nothing changes. And as I said, I'll be rebooting the f-ing things this weekend.
~ Reboot now! No time for the weekend.. If it's gonna break its gonna break on company time damn it! ~
-
@DustinB3403 said in Xenserver Space Woes:
~ Reboot now! No time for the weekend.. If it's gonna break its gonna break on company time damn it! ~
Well it's not broken now, everything is working as expected. So if I can schedule the down time it'll be better for everyone. However I am getting a "Run out of space while coalescing," which come to think on it may also be related...
-
@jrc said in Xenserver Space Woes:
@DustinB3403 said in Xenserver Space Woes:
~ Reboot now! No time for the weekend.. If it's gonna break its gonna break on company time damn it! ~
Well it's not broken now, everything is working as expected. So if I can schedule the down time it'll be better for everyone. However I am getting a "Run out of space while coalescing," which come to think on it may also be related...
Oh man... You just messed up. Now it is definitely going to break!
