Someone doesn't like local storage for large amounts of data
-
This whole topic came up under the discussion about Continuous Replication.
Dustin wants to replicate his VMs from his primary to a secondary. His main VM is a filer it has many (like 4-7 large - 500 GB+ ) volumes.
From what I can tell that @olivier was suggesting that the 'data' of this server should be elsewhere, not on the VM host. Of course the data could be on a SAN/NAS/Other VM host, whatever, but @olivier was suggesting that it shouldn't be on the XS in question because it would be to big and slow. OK fine - but Dustin still wants to protect his data, and make it have very small downtime windows. So, if Dustin puts the data portion on another XS box - Dustin would still need/want to CR that host to another host allowing him to spin up that data very quickly in the case of a failure.
So in the stated above case, what benefit is there in putting the data any place else other than on the VM - because no matter what, Dustin wants the full data in two places (live and near live).
Does this make sense?
-
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@FATeknollogee said in Someone doesn't like local storage for large amounts of data:
Isn't this 2016 or did I read my calendar wrong?
Are you surprised that enormous files are a problem?
No, I am not surprised at the file size, I'm a 'lil baffled that you want to use a raw partition
-
@FATeknollogee said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@FATeknollogee said in Someone doesn't like local storage for large amounts of data:
Isn't this 2016 or did I read my calendar wrong?
Are you surprised that enormous files are a problem?
No, I am not surprised at the file size, I'm a 'lil baffled that you want to use a raw partition
But aren't the two things one and the same? If you have a limitation on file sizes, raw is the only other option.
-
@scottalanmiller Isn't that part of why we all virtualize so we don't have deal with raw?
-
Dustin would still need/want to CR that host to another host allowing him to spin up that data very quickly in the case of a failure.
Use tools built for that. GFS2, Gluster, Ceph, Swift, Cinder, etc. The VM would remount after booting in the new host and the storage still fails over.
-
@FATeknollogee said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller Isn't that part of why we all virtualize so we don't have deal with raw?
No, the idea of putting local storage into files is a post-virtualization concept.
-
@FATeknollogee said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller Isn't that part of why we all virtualize so we don't have deal with raw?
You can still snapshot raw. Raw can be an image file or a volume or a full disk. Raw doesn't mean not virtualized.
-
@stacksofplates said in Someone doesn't like local storage for large amounts of data:
Dustin would still need/want to CR that host to another host allowing him to spin up that data very quickly in the case of a failure.
Use tools built for that. GFS2, Gluster, Ceph, Swift, Cinder, etc. The VM would remount after booting in the new host and the storage still fails over.
Agreed. The problem that is being run into here is one of replication capacity and affects a NAS the same that affects a VM. So you solve both in the same way.
In a VM, you turn to Gluster, et al. In physical you turn to Exablox or similar.
-
@stacksofplates said in Someone doesn't like local storage for large amounts of data:
@FATeknollogee said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller Isn't that part of why we all virtualize so we don't have deal with raw?
You can still snapshot raw. Raw can be an image file or a volume or a full disk. Raw doesn't mean not virtualized.
ANd we did, a lot, prior to virtualizing.
-
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@stacksofplates said in Someone doesn't like local storage for large amounts of data:
@FATeknollogee said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller Isn't that part of why we all virtualize so we don't have deal with raw?
You can still snapshot raw. Raw can be an image file or a volume or a full disk. Raw doesn't mean not virtualized.
ANd we did, a lot, prior to virtualizing.
Ya I still do for our workstations.
-
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@stacksofplates said in Someone doesn't like local storage for large amounts of data:
Dustin would still need/want to CR that host to another host allowing him to spin up that data very quickly in the case of a failure.
Use tools built for that. GFS2, Gluster, Ceph, Swift, Cinder, etc. The VM would remount after booting in the new host and the storage still fails over.
Agreed. The problem that is being run into here is one of replication capacity and affects a NAS the same that affects a VM. So you solve both in the same way.
In a VM, you turn to Gluster, et al. In physical you turn to Exablox or similar.
We have two Isilons coming. One is here and ready to be installed. Much easier than managing all of that myself.
Our guys can generate about 20TB a week between them all.
-
@stacksofplates said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@stacksofplates said in Someone doesn't like local storage for large amounts of data:
Dustin would still need/want to CR that host to another host allowing him to spin up that data very quickly in the case of a failure.
Use tools built for that. GFS2, Gluster, Ceph, Swift, Cinder, etc. The VM would remount after booting in the new host and the storage still fails over.
Agreed. The problem that is being run into here is one of replication capacity and affects a NAS the same that affects a VM. So you solve both in the same way.
In a VM, you turn to Gluster, et al. In physical you turn to Exablox or similar.
We have two Isilons coming. One is here and ready to be installed. Much easier than managing all of that myself.
Our guys can generate about 20TB a week between them all.
Looked at Isilon a bit a few weeks ago. Definitely nice gear there.
-
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@stacksofplates said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@stacksofplates said in Someone doesn't like local storage for large amounts of data:
Dustin would still need/want to CR that host to another host allowing him to spin up that data very quickly in the case of a failure.
Use tools built for that. GFS2, Gluster, Ceph, Swift, Cinder, etc. The VM would remount after booting in the new host and the storage still fails over.
Agreed. The problem that is being run into here is one of replication capacity and affects a NAS the same that affects a VM. So you solve both in the same way.
In a VM, you turn to Gluster, et al. In physical you turn to Exablox or similar.
We have two Isilons coming. One is here and ready to be installed. Much easier than managing all of that myself.
Our guys can generate about 20TB a week between them all.
Looked at Isilon a bit a few weeks ago. Definitely nice gear there.
For the price it should be.
-
@stacksofplates Oh yeah, we didn't go with it, not cost effective at all. The price was a bit crazy.
-
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@stacksofplates Oh yeah, we didn't go with it, not cost effective at all. The price was a bit crazy.
We shaved it down a bit by supplying our own rack, power cables, PDU, etc. They tried to throw all of that in the quote.
Power cables at ~$60 a piece adds up.
-
@stacksofplates said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@stacksofplates said in Someone doesn't like local storage for large amounts of data:
Dustin would still need/want to CR that host to another host allowing him to spin up that data very quickly in the case of a failure.
Use tools built for that. GFS2, Gluster, Ceph, Swift, Cinder, etc. The VM would remount after booting in the new host and the storage still fails over.
Agreed. The problem that is being run into here is one of replication capacity and affects a NAS the same that affects a VM. So you solve both in the same way.
In a VM, you turn to Gluster, et al. In physical you turn to Exablox or similar.
We have two Isilons coming. One is here and ready to be installed. Much easier than managing all of that myself.
Our guys can generate about 20TB a week between them all.
There pretty nice. We'd gotten Demo units. We use the VMAX though.
-
@stacksofplates said in Someone doesn't like local storage for large amounts of data:
@FATeknollogee said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller Isn't that part of why we all virtualize so we don't have deal with raw?
You can still snapshot raw. Raw can be an image file or a volume or a full disk. Raw doesn't mean not virtualized.
@stacksofplates said in Someone doesn't like local storage for large amounts of data:
@FATeknollogee said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller Isn't that part of why we all virtualize so we don't have deal with raw?
You can still snapshot raw. Raw can be an image file or a volume or a full disk. Raw doesn't mean not virtualized.
RAWs biggest limitation is you can't storage vmotion it. You can vmotion the pointer but if you are retiring a SAN or something you will be doing it manually.
-
@Jason said in Someone doesn't like local storage for large amounts of data:
@stacksofplates said in Someone doesn't like local storage for large amounts of data:
@FATeknollogee said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller Isn't that part of why we all virtualize so we don't have deal with raw?
You can still snapshot raw. Raw can be an image file or a volume or a full disk. Raw doesn't mean not virtualized.
@stacksofplates said in Someone doesn't like local storage for large amounts of data:
@FATeknollogee said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller Isn't that part of why we all virtualize so we don't have deal with raw?
You can still snapshot raw. Raw can be an image file or a volume or a full disk. Raw doesn't mean not virtualized.
RAWs biggest limitation is you can't storage vmotion it. You can vmotion the pointer but if you are retiring a SAN or something you will be doing it manually.
Ya I'm using KVM so I can either move the raw file or the volume to wherever I need it. It's really simple with qemu and lvm.
