Examples of proper utilization of SAN



  • @Pete-S said in Examples of proper utilization of SAN:

    If you look at it a SAN is very desirable when you're running large databases, typical of the enterprise, because you need low latency block storage.

    SAN always increases latency. I'd love it to be magic, but whenever you add more connections, you add latency. There is no getting around it. If you need low latency, you always go local.



  • @Pete-S said in Examples of proper utilization of SAN:

    If you look at it a SAN is very desirable when you're running large databases, typical of the enterprise

    The part quoted is the only bit that makes sense.

    More devices, more LAN, more connections, all external storage means it will always be slower. If you could get a single massive server with that much storage, you'd out perform a SAN every day all day.



  • SAN only make sense when you have massive storage requirements, tens of hundreds of Terabytes and upward.



  • @Pete-S said in Examples of proper utilization of SAN:

    If you look at it a SAN is very desirable when you're running large databases, typical of the enterprise, because you need low latency block storage.

    SANs using all flash NVMe arrays have very impressive performance and a price to match of course. For it to make economic sense you have to have the need for that kind of storage and speed and I guess that's why it falls into the enterprise market and not SMBs. If most SMBs can run their entire infrastructure on two or three server it's hard to see how they would have the need for a SAN - unless they are some kind of provider.

    What's wrong with NVMe in a non-San shared nothing setup, or even a serverless database?



  • @Obsolesce said in Examples of proper utilization of SAN:

    What's wrong with NVMe in a non-San shared nothing setup, or even a serverless database?

    What?

    There is nothing wrong with it, besides that it doesn't scale easily.

    Edit: Also it doesn't fit the subject of "proper utilization of SAN", because that's just a standalone server.



  • @travisdh1 said in Examples of proper utilization of SAN:

    @Pete-S said in Examples of proper utilization of SAN:

    If you look at it a SAN is very desirable when you're running large databases, typical of the enterprise, because you need low latency block storage.

    SAN always increases latency. I'd love it to be magic, but whenever you add more connections, you add latency. There is no getting around it. If you need low latency, you always go local.

    When I wrote "large database" I didn't talk about a wordpress installation. So it's implied that if we are talking about SAN we are talking about shared block storage - meaning local storage is out.



  • @Pete-S said in Examples of proper utilization of SAN:

    @travisdh1 said in Examples of proper utilization of SAN:

    @Pete-S said in Examples of proper utilization of SAN:

    If you look at it a SAN is very desirable when you're running large databases, typical of the enterprise, because you need low latency block storage.

    SAN always increases latency. I'd love it to be magic, but whenever you add more connections, you add latency. There is no getting around it. If you need low latency, you always go local.

    When I wrote "large database" I didn't talk about a wordpress installation. So it's implied that if we are talking about SAN we are talking about shared block storage - meaning local storage is out.

    Talking about SAN at all means local storage is out. Why are yourself and @Obsolesce talking about local storage at all?

    The only reasonable use case for SAN is with massive scale out storage requirements.



  • Talking about vSAN, which this topic very clearly isn't based on:

    @EddieJennings said in Examples of proper utilization of SAN:

    Yes. I'm intending the scope of the discussion to be about a physical SAN (storage devices and their network).

    Would of course take advantage of the local storage on each server and create a SAN with it. But it still functions as a SAN.

    So talking about, presumably things like Dell EMC SAN products (throw a name up) there is no reasonable need for them unless you are discussing scale out storage requirements.



  • @EddieJennings what conversation is going on that you're looking for more information regarding SAN (products I assume). Which SAN isn't something you can purchase, it's something you have to build.



  • @DustinB3403 said in Examples of proper utilization of SAN:

    The only reasonable use case for SAN is with massive scale out storage requirements.

    Wrong. Low latency shared block storage for OLTP applications don't have to be massive to make sense. Just need high performance requirements. Also, for instance a HPC cluster might fit in one rack but need a high performance storage solution.



  • Just Google: When to Consider a SAN

    A voila, first hit.



  • @DustinB3403 said in Examples of proper utilization of SAN:

    Large scale out storage is the only logical use. Storage way above what could be fit in a single server.

    SAN is scale up, not scale out.



  • @Dashrender said in Examples of proper utilization of SAN:

    @DustinB3403 said in Examples of proper utilization of SAN:

    @davide-bonavita said in Examples of proper utilization of SAN:

    We deployed a starwind vSAN in HA to store some critical VMs, it works quite well (cfr. "StarWind Virtual SAN®
    Installation and Configuration of HyperConverged 2 Nodes with Hyper-V Cluster" technical paper)

    While that is an good example of uses to deploy a SAN solution, I think @EddieJennings is referring to physical SAN products and not the logical vSAN solutions that we know about today.

    OK that brings up a good point - is physical SANs even really worth it much anymore today considering the abilities of vSANs? I mean I'm sure there are times where it can be worthwhile - but likely not for anyone really hanging out on these forums.

    vSANs were there first. Physical SAN came later, by definition. We get all weird when talking about SANs, but in the real world, everything is software first and appliance later. NAS is an appliance of a file server. SAN is an appliance of a block storage server. SAN became "so famous" and so treated as a magic black box, that people had to go back and rename the original product a vSAN so that people would know what it was. Would be the same as calling a normal file server a vNAS today. Sounds stupid, but that's how stupid vSAN is.

    We've never had a time when vSAN wasn't everywhere.



  • @Pete-S said in Examples of proper utilization of SAN:

    If you look at it a SAN is very desirable when you're running large databases, typical of the enterprise, because you need low latency block storage.

    Actually that's where you avoid it. Specifically for that reason. Because for databases the additional latency and risk of the SAN doesn't normally make sense. That's why in high performance databases were one of the first places to abandon SAN because they needed something faster.

    Remember, SAN is the slow option, not the fast one. Simple physics says that a SAN has to be slower than its local equivalent. Maybe not a lot, but it's physically impossible for it to be as fast or faster. It has to be at least a tiny bit slower. SAN is always chosen despite performance losses.



  • @DustinB3403 said in Examples of proper utilization of SAN:

    @Pete-S said in Examples of proper utilization of SAN:

    If you look at it a SAN is very desirable when you're running large databases, typical of the enterprise

    The part quoted is the only bit that makes sense.

    Not really, because databases tend to need speed and reliability and they all replicate at the application layer and can't be replicated blindly at the storage layer nor is there a real use case of multiple RDBMS seeing a single pool of storage - the locking problems would be terrible for performance. So for myriad reasons, we would expect databases to be among the worst use cases for SAN. And in the real world, that's exactly where we saw SANs avoided first. Databases were where we exposed the needs for local storage options first.

    Of course, the problem is, most places just throw money at solutions until even the worst option works. Then people who see that working assume it was a good choice, instead of a bad one, because they see it "in use" rather than see the evaluation, cost and risk involved.



  • @Pete-S said in Examples of proper utilization of SAN:

    So it's implied that if we are talking about SAN we are talking about shared block storage - meaning local storage is out.

    Local storage is the best way to share block storage. In no way whatsoever does needing shared block imply that a SAN is a need.

    https://smbitjournal.com/2013/07/replicated-local-storage/

    Much of the worst runs on shared local. Some do it by using block protocols like a vSAN, some do it without through things like Gluster or SCRIBE. But RLS is the best way to get high performance shared block, if you need shared.



  • @Pete-S said in Examples of proper utilization of SAN:

    @DustinB3403 said in Examples of proper utilization of SAN:

    The only reasonable use case for SAN is with massive scale out storage requirements.

    Wrong. Low latency shared block storage for OLTP applications don't have to be massive to make sense. Just need high performance requirements. Also, for instance a HPC cluster might fit in one rack but need a high performance storage solution.

    As Travis had pointed out, SAN guarantees more latency, not less. So any low latency requirements make SAN less desirable, not more. Your belief that SAN provides more performance than local storage does is causing you to think SAN would solve problems where it doesn't (or doesn't as well as more obvious solutions.)

    This is basic computing physics.... the same storage local or distant has to be faster when local. Maybe not much faster, but it can't be slower or the same. There is just less latency - less wire, fewer hops.

    Using SAN implies only two things.... distant, and block. Anything else assumed about SAN is just incorrect, it's not part of SAN.

    How most people approach it is that they assume their SAN is crazy expensive and their local is cheap and then use that to show that SAN is "faster" by comparing apples and oranges. But the same NVMe drive local vs. hooked to a separate server and shared over even NVMeoF is a tiny bit slower.



  • @scottalanmiller said in Examples of proper utilization of SAN:

    @Pete-S said in Examples of proper utilization of SAN:

    @DustinB3403 said in Examples of proper utilization of SAN:

    The only reasonable use case for SAN is with massive scale out storage requirements.

    Wrong. Low latency shared block storage for OLTP applications don't have to be massive to make sense. Just need high performance requirements. Also, for instance a HPC cluster might fit in one rack but need a high performance storage solution.

    As Travis had pointed out, SAN guarantees more latency, not less. So any low latency requirements make SAN less desirable, not more. Your belief that SAN provides more performance than local storage does is causing you to think SAN would solve problems where it doesn't (or doesn't as well as more obvious solutions.)

    This is basic computing physics.... the same storage local or distant has to be faster when local. Maybe not much faster, but it can't be slower or the same. There is just less latency - less wire, fewer hops.

    Using SAN implies only two things.... distant, and block. Anything else assumed about SAN is just incorrect, it's not part of SAN.

    How most people approach it is that they assume their SAN is crazy expensive and their local is cheap and then use that to show that SAN is "faster" by comparing apples and oranges. But the same NVMe drive local vs. hooked to a separate server and shared over even NVMeoF is a tiny bit slower.

    You assumed I made assumptions I didn't make.

    Yes, local is always faster but local is not shared. So then it all becomes just a question how we share and access the data. If we put the shared storage on dedicated servers we have a SAN. If we put the storage on the same servers that we are running compute we have hyperconverged storage.

    In the first case we can optimize both hardware and software and it's only running this single task. In the second case we usually run both compute and storage on the same hardware. By pure logic the first case, the SAN, has to be the higher performing option.

    Looking at replicated local storage though, that implies that we can fit the storage on one server. Of course this is almost as fast as local storage (assuming synchronous replication). But it also means that SAN advantage of consolidating storage is lost.

    So in order of performance on equal hardware we have:

    1. local storage
    2. replicated local storage
    3. SAN
    4. vSAN and similar

    Of course then we have local storage cache for vSAN solutions and other things to mess this up. Also of course in real life its the cost of it all that determines what is the best solution.

    When I said that the SAN is the low latency option for OLTP or HPC, it's compared to things like gluster or vSAN - as they are comparable when it comes to storage consolidation. But you need enough workloads and servers for it to make sense to consolidate. Consolidate increase utilization (lower cost) by sacrificing some performance and increasing complexity.



  • @Pete-S said in Examples of proper utilization of SAN:

    Yes, local is always faster but local is not shared

    But local CAN be shared. SAN is not shared either, but CAN be shared. Both are both shared or not shared.



  • @Pete-S said in Examples of proper utilization of SAN:

    In the first case we can optimize both hardware and software and it's only running this single task. In the second case we usually run both compute and storage on the same hardware. By pure logic the first case, the SAN, has to be the higher performing option.

    That is in no way logical. That actually is both incredibly unlikely due to logic, and totally not true in the real world. I have no idea what kind of logic would make you think that making them far away from each other would be fast and local slow based on "dedicated harware" when the resource needs of storage is so tiny that it's of no consequence in modern devices.



  • @scottalanmiller said in Examples of proper utilization of SAN:

    @Pete-S said in Examples of proper utilization of SAN:

    Yes, local is always faster but local is not shared

    But local CAN be shared. SAN is not shared either, but CAN be shared. Both are both shared or not shared.

    Local or not local has to be a question of where the data is used. It's always local somewhere.



  • @Pete-S said in Examples of proper utilization of SAN:

    Looking at replicated local storage though, that implies that we can fit the storage on one server. Of course this is almost as fast as local storage (assuming synchronous replication). But it also means that SAN advantage of consolidating storage is lost.

    RLS does imply that, yes. But SAN does, too. SAN and RLS both have the "fit it in one server" limitation.

    In almost all cases, you need SAN to replicate, too. So any replication overhead in 99.9% of cases is the same RLS vs SAN. If your SAN doesn't need to be replicated, then chances are your local storage does not. There are extreme cases where you need shared storage that isn't replicated for reliability where SAN has a consolidation advantage for lower criticality workloads.



  • @Pete-S said in Examples of proper utilization of SAN:

    @scottalanmiller said in Examples of proper utilization of SAN:

    @Pete-S said in Examples of proper utilization of SAN:

    Yes, local is always faster but local is not shared

    But local CAN be shared. SAN is not shared either, but CAN be shared. Both are both shared or not shared.

    Local or not local has to be a question of where the data is used. It's always local somewhere.

    Yes, local to the computer or local to a remote dedicated storage server.

    RLS means that the data is LOCAL to multiple locations.



  • @Pete-S said in Examples of proper utilization of SAN:

    So in order of performance on equal hardware we have:

    local storage
    replicated local storage
    SAN
    vSAN and similar

    I would not break it down that way. RLS can be as fast as any other local storage, if you don't require full sync. Async is an option and can have no performance overhead.

    vSAN is not slower than SAN. A SAN and vSAN are the same speed. And in the real world, since vSAN options are more flexible, they are actually faster.



  • @Pete-S said in Examples of proper utilization of SAN:

    When I said that the SAN is the low latency option for OLTP or HPC, it's compared to things like gluster or vSAN - as they are comparable when it comes to storage consolidation

    I see. The problem with that is that Gluster is specifically a super slow mechanism. It's not that it is RLS that makes it slow, it's the Gluster mechanism itself. It's just a solution not designed around speed. So yes, it's slow. But the Netgear SC101 SAN is way slower, even though it is a real SAN on dedicated hardware.

    If we look at the faster SAN and RLS options, like a top end EMC vs. a Starwind vSAN then we are at blinding speeds in both cases with, I believe, the Starwind pulling ahead with things like NVMeoF that is pretty much as fast as things get, and RAM cache replication over RDMA on 100Mb/s Infiniband.



  • @scottalanmiller said in Examples of proper utilization of SAN:

    @Pete-S said in Examples of proper utilization of SAN:

    When I said that the SAN is the low latency option for OLTP or HPC, it's compared to things like gluster or vSAN - as they are comparable when it comes to storage consolidation

    I see. The problem with that is that Gluster is specifically a super slow mechanism. It's not that it is RLS that makes it slow, it's the Gluster mechanism itself. It's just a solution not designed around speed. So yes, it's slow. But the Netgear SC101 SAN is way slower, even though it is a real SAN on dedicated hardware.

    If we look at the faster SAN and RLS options, like a top end EMC vs. a Starwind vSAN then we are at blinding speeds in both cases with, I believe, the Starwind pulling ahead with things like NVMeoF that is pretty much as fast as things get, and RAM cache replication over RDMA on 100Mb/s Infiniband.

    But look at next-gen SANs like Pure Storage and Netapp's NVMe arrays also running NVMe over Fabric. I think they are well ahead.

    I don't know how Starwind vSAN can be run but if it's on a hypervisor it's severely limited by I/O congestion through the kernel. NVMe drives is causing problems that was of no concern whatsoever with spinners. Both KVM and Xen has made a lot of work to limit their I/O latency and use polling techniques now but it's still a problem. That's why you really need SR-IOV on NVMe drives so any VM can bypass the hypervisor and just have it's own kernel to slow things down.



  • @Pete-S said in Examples of proper utilization of SAN:

    @scottalanmiller said in Examples of proper utilization of SAN:

    @Pete-S said in Examples of proper utilization of SAN:

    When I said that the SAN is the low latency option for OLTP or HPC, it's compared to things like gluster or vSAN - as they are comparable when it comes to storage consolidation

    I see. The problem with that is that Gluster is specifically a super slow mechanism. It's not that it is RLS that makes it slow, it's the Gluster mechanism itself. It's just a solution not designed around speed. So yes, it's slow. But the Netgear SC101 SAN is way slower, even though it is a real SAN on dedicated hardware.

    If we look at the faster SAN and RLS options, like a top end EMC vs. a Starwind vSAN then we are at blinding speeds in both cases with, I believe, the Starwind pulling ahead with things like NVMeoF that is pretty much as fast as things get, and RAM cache replication over RDMA on 100Mb/s Infiniband.

    But look at next-gen SANs like Pure Storage and Netapp's NVMe arrays also running NVMe over Fabric. I think they are well ahead.

    I don't know how Starwind vSAN can be run but if it's on a hypervisor it's severely limited by I/O congestion through the kernel. NVMe drives is causing problems that was of no concern whatsoever with spinners. Both KVM and Xen has made a lot of work to limit their I/O latency and use polling techniques now but it's still a problem. That's why you really need SR-IOV on NVMe drives so any VM can bypass the hypervisor and just have it's own kernel to slow things down.

    Anyway, I'm sure every vendor is working on making things faster all the time. There are just to many option to make blanket statements on anything without being specific. For any discussion like this you'd have to have some idea of exactly what you want to accomplish and what the budget is to determine if option A or option B is the best course of action.



  • @scottalanmiller said in Examples of proper utilization of SAN:

    Just Google: When to Consider a SAN

    A voila, first hit.

    Reading that is what prompted me to make this thread. I am looking for examples of $enterprises running $applications which requires infrastructure that would necessitate the use of SAN.

    To efficiently leverage consolidation it is necessary to have scale and this is where SANs really shine – when scale but in capacity and, more importantly, in the number of attaching nodes become very large. SANs are best suited to large scale storage consolidation. This is their sweet spot and what makes them nearly ubiquitous in large enterprises and very rare in small ones.

    I'm trying to of examples of where this situation exists. Would this be a valid example? Take Vultr, who's providing VPS services. Because of the number of hosts hosting the VMs which are eventually used by customers for their various projects, the best way to present block storage to these hosts is from a SAN.



  • @DustinB3403 said in Examples of proper utilization of SAN:

    @EddieJennings what conversation is going on that you're looking for more information regarding SAN (products I assume). Which SAN isn't something you can purchase, it's something you have to build.

    Been busy this morning, and haven't been able to follow the thread closely. I know that SAN is something that's built, and what I was looking for are real-world examples of infrastructures that utilize SAN and see what about their needs leads to the decision that building a SAN is necessary.

    Likely there isn't any kind of clear example of the process that leads to said decision, but at least it got folks talking, so I can go back in a bit, read what's there, and see if there's wisdom to gain :).



  • @EddieJennings said in Examples of proper utilization of SAN:

    @DustinB3403 said in Examples of proper utilization of SAN:

    @EddieJennings what conversation is going on that you're looking for more information regarding SAN (products I assume). Which SAN isn't something you can purchase, it's something you have to build.

    Been busy this morning, and haven't been able to follow the thread closely. I know that SAN is something that's built, and what I was looking for are real-world examples of infrastructures that utilize SAN and see what about their needs leads to the decision that building a SAN is necessary.

    Likely there isn't any kind of clear example of the process that leads to said decision, but at least it got folks talking, so I can go back in a bit, read what's there, and see if there's wisdom to gain :).

    Good luck wading through all the straw-man arguments and examples lol.