Announcing the Death of RAID



  • Friends, IT Pros, countrymen, lend me your ears;
    I come to bury RAID, not to praise it.
    The evil that tech does lives after them;
    The good is oft interred with its bones;
    So let it be with RAID. The noble S.A.M.
    Hath told you RAID was ambitious:
    If it were so, it was a grievous fault,
    And grievously hath RAID answer’d it.

    Few people have written as extensively on RAID as I have, likely none have written so much and for so long. But we must address the fact that no matter how attached to it we are, no matter how much we talk about it or have studied it... RAID was never meant to last forever and sometime, in the last few years, while none of us were keeping watch: RAID slipped quietly away in its sleep. There were no death throws, no loud bangs, no sudden heart attack moment. Just peacefully, at home alone with the lights off RAID left us to go on to the big technology bin in the sky.

    So what exactly happen to our beloved storage technology? The world changed around it.

    When RAID was new, nearly everything in IT was expensive, really expensive. RAID was built around a need to accommodate large numbers of small disks in single enclosures. These were the realities of the age and even networking was far from ubiquitous at the time. And the idea that storage could pass over a network was nascent.

    Today so much is different. Drives are huge, prices have changed dramatically, we rarely work from single enclosure systems and storage networking is very common and works great.

    So first the basics: scale. RAID works great for eight disks used locally in an array big enough for local applications to run. It's perfect for that. But start pushing 40TB of storage or huge spindle counts and RAID starts to have all kinds of complications. The biggest problem is that parity based RAID options (the RAID F family of R4, 5, 6 and 7) don't scale well and two RAID levels there are so pointless at scale that they have been effective retired completely (R4 and R5) and R6 only works well at smaller scales and R7 is not widely implemented and has major rebuild problems because parity just does not scale that well.

    In order combat scaling issues with parity RAID, we need to turn to mirrored RAID which lacks the capacity efficiency of parity RAID (in most cases) which leaves RAID open to "attack" at a business level from other approaches.

    So scale has simply worked against RAID to a point to where RAID just rarely makes sense outside of small, special case deployments or for limited use like just for local boot disks.

    The move from single enclosures to clusters. Outside of storage itself, businesses have broadly moved from running single enclosure systems (one server, on its own) to clusters of servers often in a virtualization stack. This move has made it critical that our storage be able to be accessed by multiple systems at once over the network and presents an opportunity for data protection between nodes (internodal reliability rather than purely intranodal reliability.)

    We can address this to some degree with networked RAID (products like DRBD) but this does not scale well (or often at all) past two nodes and presents large problems - such as a need to consider node reliability and rebuilds both individually and together and how reads and writes will be impacted by the network. RAID quickly becomes very complex to implement.


    Because systems where RAID fails to offer a straightforward, elegant solution have moved from niche to the norm, RAID itself now faces challenges that appear to be more than it will be able to withstand.

    Enter RAIN. RAIN is really a catch all for many approaches to dealing with the RAID deficiency problems. But common approaches are beginning to emerge that are giving rise to many advantageous RAIN implementations.

    RAIN refers to a "Redundant Array of Independent Nodes" and is a play on the RAID term. The RAIN term itself is too loose to be useful. RAIN's name only implies network RAID, but that is not what is meant by its use.

    For a system to be generally accepted as RAIN, rather than Network RAID, it should be both node and drive aware and provide data protection (unlike technologies like RAID 0.) This awareness allows RAIN approaches to place data on disk in such a way as to protect against the failure of any one disk and/or any one node.

    The most common or popular implementations of RAIN rely on disk mirroring, much akin to RAID 10, but doing so at a more granular level such as the block, rather than at a full disk or logical disk level. Mirroring has become the standard due to the low cost of disks today combined with the common advantages of mirroring, mostly around reliability and speed.

    RAIN addresses the large scale reliability issues of RAID by making failure domains much smaller (individual blocks commonly) and leveraging mirroring which scales very well. RAIN additionally addresses scalability by allowing a single storage pool to scale far past the size of a single enclosure.

    RAIN addresses the networking problem by having the entire storage stack by network aware, rather than simply tunneling existing RAID implementations over a network connection. RAIN can make choices such as only reading data from the local disk rather than waiting for part of the data to be transferred over the network and can make choices such as not waiting for remote writes before continuing with write confirmation. RAIN provides more flexibility and more choices.

    I will explore RAIN technology, products and the future in another article. The future is not bleak, it is very good. The post-RAID world is not a bad one and RAID will, most likely, become mostly an anecdotal piece of IT history outside of a lingering use of RAID 1 for special cases. RAID is already niche today in larger businesses and this trend is pushing more and more into the SMB space. We can't expect everything that we do in IT to stay the same and RAID has had a good, long run as the top dog in server storage. But the times they are a changing and RAID should no longer be the assumed storage answer that it has been for so long.



  • Some RAIN vendor you may or may not know:

    • Open Source RAIN Options: Gluster, Lustre, CEPH, Swift
    • Closed Source RAIN Software: AetherStore
    • RAIN Storage Appliance: Exablox
    • RAIN Used Under the Hood: Scale HC3, VMware VSAN


  • Is StarWinds vSAN considered RAIN?



  • @coliver said in Announcing the Death of RAID:

    Is StarWinds vSAN considered RAIN?

    We'd have to dig in under the hood. I think that they are mostly focused on network RAID, just really advanced.



  • @scottalanmiller said in Announcing the Death of RAID:

    RAIN addresses the networking problem by having the entire storage stack by network aware, rather than simply tunneling existing RAID implementations over a network connection.

    I think you mean be



  • What's the smallest install where you would consider RAIN instead of RAID? and if that isn't enough details, what additional details are needed before you could answer a question like that?



  • @Dashrender said in Announcing the Death of RAID:

    What's the smallest install where you would consider RAIN instead of RAID? and if that isn't enough details, what additional details are needed before you could answer a question like that?

    When you have more than one server πŸ™‚



  • RAID and RAIN have overlap in the two node world. But here is a quick breakdown of which makes the most sense where.

    • Single Server: RAID
    • Dual Server Cluster: Network RAID or RAIN
    • Three or More Server Cluster: RAIN

    It's the number of nodes in a related cluster (shared storage) that makes the difference. And RAIN may encourage systems that were not clustered before to become clustered, as well.



  • RAIN can be scaled down to a single node, but you lose the "requirement" of nodal protection. So we could think of it as "RAIN 0" or a RAIN in a failed state. Examples of this include the single node Scale HC3 product and the single node Exablox implementation.



  • @scottalanmiller said in Announcing the Death of RAID:

    RAIN can be scaled down to a single node, but you lose the "requirement" of nodal protection. So we could think of it as "RAIN 0" or a RAIN in a failed state. Examples of this include the single node Scale HC3 product and the single node Exablox implementation.

    Right... but ideally to convert a single node RAIN instance to a two-node RAIN cluster, you only need to add a second host and appropriate storage, right?



  • @dafyre said in Announcing the Death of RAID:

    @scottalanmiller said in Announcing the Death of RAID:

    RAIN can be scaled down to a single node, but you lose the "requirement" of nodal protection. So we could think of it as "RAIN 0" or a RAIN in a failed state. Examples of this include the single node Scale HC3 product and the single node Exablox implementation.

    Right... but ideally to convert a single node RAIN instance to a two-node RAIN cluster, you only need to add a second host and appropriate storage, right?

    Depends on if you are addressing the need for a witness node. In theory, yes two node can do RAIN. But it comes with complications.



  • @scottalanmiller said in Announcing the Death of RAID:

    @dafyre said in Announcing the Death of RAID:

    @scottalanmiller said in Announcing the Death of RAID:

    RAIN can be scaled down to a single node, but you lose the "requirement" of nodal protection. So we could think of it as "RAIN 0" or a RAIN in a failed state. Examples of this include the single node Scale HC3 product and the single node Exablox implementation.

    Right... but ideally to convert a single node RAIN instance to a two-node RAIN cluster, you only need to add a second host and appropriate storage, right?

    Depends on if you are addressing the need for a witness node. In theory, yes two node can do RAIN. But it comes with complications.

    OK considering the witness role, are you saying it's mostly wise to not worry about RAIN until you reach 3+ nodes?



  • @Dashrender said in Announcing the Death of RAID:

    @scottalanmiller said in Announcing the Death of RAID:

    @dafyre said in Announcing the Death of RAID:

    @scottalanmiller said in Announcing the Death of RAID:

    RAIN can be scaled down to a single node, but you lose the "requirement" of nodal protection. So we could think of it as "RAIN 0" or a RAIN in a failed state. Examples of this include the single node Scale HC3 product and the single node Exablox implementation.

    Right... but ideally to convert a single node RAIN instance to a two-node RAIN cluster, you only need to add a second host and appropriate storage, right?

    Depends on if you are addressing the need for a witness node. In theory, yes two node can do RAIN. But it comes with complications.

    OK considering the witness role, are you saying it's mostly wise to not worry about RAIN until you reach 3+ nodes?

    Generally, yes. Nothing wrong with it at one node, just not very much value to it there. And at two nodes, no one is focused on making that work well, it's just niche and unimportant.



  • @scottalanmiller said in Announcing the Death of RAID:

    • Closed Source RAIN Software: AetherStore

    It doesn't seem like you can use AetherStore for VM storage?



  • @FATeknollogee said in Announcing the Death of RAID:

    @scottalanmiller said in Announcing the Death of RAID:

    • Closed Source RAIN Software: AetherStore

    It doesn't seem like you can use AetherStore for VM storage?

    Hrm, is that a challenge? If it can be mounted on the hypervisor, it can be used as storage. Now I want to go test, and I just don't have the time!



  • I just looked at the website use cases & it didn't mention anything about virtualization, so I assumed it couldn't be used.



  • @FATeknollogee said in Announcing the Death of RAID:

    @scottalanmiller said in Announcing the Death of RAID:

    • Closed Source RAIN Software: AetherStore

    It doesn't seem like you can use AetherStore for VM storage?

    You CAN, but you sure wouldn't.



  • @travisdh1 said in Announcing the Death of RAID:

    @FATeknollogee said in Announcing the Death of RAID:

    @scottalanmiller said in Announcing the Death of RAID:

    • Closed Source RAIN Software: AetherStore

    It doesn't seem like you can use AetherStore for VM storage?

    Hrm, is that a challenge? If it can be mounted on the hypervisor, it can be used as storage. Now I want to go test, and I just don't have the time!

    It's a block device, it has to work. Would just be awful.



  • @FATeknollogee said in Announcing the Death of RAID:

    I just looked at the website use cases & it didn't mention anything about virtualization, so I assumed it couldn't be used.

    Just not a good use case for it. It's a block device, which should tell you everything that you need to know. It can be used for any block device task - including building a SAN on top of it.



  • @scottalanmiller
    A SAN built on 5400-7200 rpm spindles(probably) of varying size/cache, over a network with all your other traffic on it.
    Huge performance.



  • @momurda said in Announcing the Death of RAID:

    @scottalanmiller
    A SAN built on 5400-7200 rpm spindles(probably) of varying size/cache, over a network with all your other traffic on it.
    Huge performance.

    Not only that, the write speed of Aetherstore isn't fast enough to keep up with running VMs.



  • @momurda said in Announcing the Death of RAID:

    @scottalanmiller
    A SAN built on 5400-7200 rpm spindles(probably) of varying size/cache, over a network with all your other traffic on it.
    Huge performance.

    And it is not designed for speed, so even with a dedicated network and SSDs, it isn't all that fast. Not built for that.



  • @scottalanmiller nice article πŸ™‚
    Something to add to the learnings for this year πŸ™‚


  • Vendor

    @scottalanmiller said in Announcing the Death of RAID:

    @coliver said in Announcing the Death of RAID:

    Is StarWinds vSAN considered RAIN?

    We'd have to dig in under the hood. I think that they are mostly focused on network RAID, just really advanced.

    StarWind uses local reconstruction codes (for now - stand-alone software or hardware RAID on every node; can be RAID0, 1, 5, 6 or 10) and inter-node n-way replication between the nodes, can be considered as a network RAID1. There's no network parity RAID like HPE (ex-Left Hand) or Ceph does.

    P.S. We're working on our own local reconstruction codes now, so local protection (SimpliVity style) won't be required soon. FYI.



  • ...and then you have companies that cluster servers, with each server having RAID configured. Sacrificing some usable storage there.



  • @BBigford said in Announcing the Death of RAID:

    ...and then you have companies that cluster servers, with each server having RAID configured. Sacrificing some usable storage there.

    That's not uncommon and that's kinda of what Kooler is talking about, they use RAID often on individual nodes as a local means of avoiding full rebuilds under most conditions.



  • I would treat RAID as a kind of hardware offload since RAIN is known to consume more resources and thus resulting in less performance from the storage array. That is probably one of the major reasons why vendors like StarWind keep using hardware RAID. Especially on smaller deployments (storage capacities).



  • @Net-Runner said in Announcing the Death of RAID:

    I would treat RAID as a kind of hardware offload since RAIN is known to consume more resources and thus resulting in less performance from the storage array. That is probably one of the major reasons why vendors like StarWind keep using hardware RAID. Especially on smaller deployments (storage capacities).

    I wonder if this flies in the face of what @scottalanmiller has been saying that hardware RAID isn't needed for performance reasons?


  • Vendor

    @Dashrender said in Announcing the Death of RAID:

    @Net-Runner said in Announcing the Death of RAID:

    I would treat RAID as a kind of hardware offload since RAIN is known to consume more resources and thus resulting in less performance from the storage array. That is probably one of the major reasons why vendors like StarWind keep using hardware RAID. Especially on smaller deployments (storage capacities).

    I wonder if this flies in the face of what @scottalanmiller has been saying that hardware RAID isn't needed for performance reasons?

    There are many ways to skin a cat and there are some things you can't do w/out hardware components: f.e. SAS in HBA mode can't allow write cache enabled on the disks and can't enable aggressive write-back battery-protected cache because... HBA has none πŸ˜‰ This means either you acknowledge writes in DRAM (synchronized with some other hosts) or you have to use Enterprise-grade SSDs. Guys like VMware and Microsoft who claim they don't rely on hardware and you can throw away RAID cards are... cheating you! Because now you have to swap RAID cards -> Enterprise grade SSDs they can use as a cache. Pay money to save money. Sweet!



  • In other words, I think that @scottalanmiller has been saying that SMBs so rarely tax their systems so much that the performance drain put on the system by software RAID would barely be noticed. So the use of RAID as a hardware offload for RAIN wouldn't make sense - even moreso it doesn't make sense since RAIN itself is completely dependent upon the system CPU and NIC/network resources, not the RAID controller itself.

    As mentioned by @scottalanmiller above,

    @scottalanmiller said in Announcing the Death of RAID:

    ... they use RAID often on individual nodes as a local means of avoiding full rebuilds under most conditions.

    This makes sense, but will offer no performance gains on the RAIN side of the house.

    Unless I've misunderstood something.


Log in to reply