ML
    • Recent
    • Categories
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Revisiting ZFS and FreeNAS in 2019

    IT Discussion
    18
    138
    17.0k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • JaredBuschJ
      JaredBusch
      last edited by JaredBusch

      So I poked a bit on things @xrobau.
      I can't imagine FreeNAS has improved. but maybe it has.

      It is certainly not something I have considered in recent history.
      843021f1-aacb-417a-8da5-d8304ffc3208-image.png

      Also Clearly IP's current job listing for a software developer: https://docs.google.com/forms/d/e/1FAIpQLScmwk2SdunbH4RDvvbfZeWv5IAu-vxzeyTjGHZEbf5JRWVKqg/viewform

      The desirable list does not mean much. But I did notice the last line of bonus skills.
      0367fae5-4769-4c40-b89b-c5fdc049c418-image.png

      1 Reply Last reply Reply Quote 3
      • scottalanmillerS
        scottalanmiller @xrobau
        last edited by scottalanmiller

        @xrobau with years as a storage engineer, we use FreeNAS as the gauge of what "bad storage" looks like. It's not slick or good, it's so bad it's been the basis of many "how not to make software" talks. It's high risk, low benefit.

        https://smbitjournal.com/2015/07/the-jurassic-park-effect/

        https://smbitjournal.com/2014/05/the-cult-of-zfs/

        As someone called in because people got caught by the FreeNAS/ZFS hype and lost all their storage, I can tell you, this is the "red flag" of bad system design that we look for. It has none of the requirements you'd want for production usage. The parts you see as cool features are just the obvious default underlying OS and filesystem, nothing to do with FreeNAS. The parts that are dangerous are the parts that don't even matter. If you want a good storage system, ZFS and FreeBSD are fine, but you have to design HA around it. FreeNAS on its own not only doesn't do HA, it undermines it.

        FreeNAS isn't just bad as a concept, which it is - it is the most popular in a whole category that shouldn't really exist, but because of what it is, essentially all real world usages of it are really bad. It tends to be used in IPODs exacerbating the other problems that it starts out with.

        1 Reply Last reply Reply Quote 2
        • scottalanmillerS
          scottalanmiller
          last edited by scottalanmiller

          And don't forget about FreeNAS 10!!

          The release that they had to pull back and discard when they found that it was losing data all the time.

          1 Reply Last reply Reply Quote 0
          • scottalanmillerS
            scottalanmiller
            last edited by

            We even have a guide to demystifying the myths that the FreeNAS community repeats commonly:

            https://mangolassi.it/topic/16526/common-myths-we-hear-from-the-freenas-community

            xrobauX 1 Reply Last reply Reply Quote 1
            • xrobauX
              xrobau
              last edited by

              Well wow that did kinda go off on a bit of a tangent. As a bit of a rebuttal, you're kinda getting confused and claming FreeNAS is a bunch of things it isn't. In fact, in that 'common myths' page is ... I dunno, random made up stuff?

              So let's take it from MY perspective - Basically, FreeNAS is a really snazzy front end to ZFS. That's it. It ALREADY comes with a bunch of scripts to monitor SMART, resilver, validate data integrity, and send me a warm-and-fuzzy email every week saying that everything is wonderful. That's pretty much it.

              So, Why is ZFS so cool? Because it has MULTIPLE checks on data integrity. In fact, the whole design of ZFS was based around integrity, not speed. If you want fast, you don't use ZFS (or, you throw a bunch of CPU and RAM and Disks and stuff at it).

              Data Integrity is BY FAR the most important thing in data storage today. If you have a HDD with bad cache ram on it, ZFS (and, to a lesser extent, btrfs) is the only thing that's going to warn you that something is wrong with that device (and it's not like this is uncommon - I've discovered two drives in the last year that were writing corrupt data). This happens.

              So, as a Solaris administrator from way back, let's go through a couple of the misapprehensions about ZFS in that document you linked!

              • ZFS Is an Alternative to RAID - Yes. It's a DIFFERENT TYPE of what normal people think of as 'RAID' - or specifically, RAID5/RAID6. They use Parity, and when a disk is broken/missing, it does calculations to figure out the missing data. ZFS uses copies of the data. Striping and Mirroring is obviously the same.
              • "ZFS Raid Levers aren't standard" - I have no idea what that means.
              • ZFS Does not work on hardware RAID - It DOES work. You just SHOULDN'T use it with Hardware RAID. In fact, Hardware RAID is a terrible idea, and has been so for many many years now. Until hardware RAID manufacturers start putting 3ghz Xeons on their RAID cards, software raid is always going to be faster (and - more importantly - much easier to recover from a hardware failure). But the design of ZFS is that it talks to the raw spinning disks. Adding other abstraction layers in the middle is counterproductive.
              • That bit rot affects people every day and only ZFS can protect you - No, btrfs can protect you, too. Apart from that, yes - it does. As I said above, twice in about 18 months for me, and I only have about 100 spinning disks. I'm not going to get into SSDs, because I've had terrible results with a couple that wildly skew the stats - but they've burned me so badly, that the only way I'm trusting SSDs these days is in a ZRAID2 (eg, 3 copies of the data on 3 devices).
              • ... Hardware RAID is useless - Yep, these days it's a terrible idea to use hardware RAID, unless you're prepared to keep identical hardware cards with identical firmware versions in stock, right next to every machine with hardware RAID. If you haven't been bitten by a hardware RAID card being unable to import a RAID when the previous card died, you're a lucky lucky person. Also, it's slower! Why do you want to ADD unreliability, and REMOVE speed? Hardware RAID is dead. It died many years ago 😎
              • "XXX is special" - Don't know why you think someone would say that. So let's just skip over that.
              • Scrubbing is special - Yes it is! If you have a disk that is faulty, scrubbing will pick it up and repair the damage. That can only happen with ZFS, because it has COPIES of the data, and every copy has a checksum. If one of the copies doesn't match the checksum, it'll repair it by writing over it with one of the good copies.

              The other two points aren't really relevant, and don't know where they're coming from.

              So, after that nerd braindump, I am your guest ZFS expert! Ask me anything 😎

              travisdh1T scottalanmillerS maryM 15 Replies Last reply Reply Quote -5
              • xrobauX
                xrobau @scottalanmiller
                last edited by

                @scottalanmiller Oh, and I forgot one of the most important things - live compression and decompression.

                "What?" I hear you say, "why would I want to compress and decompress my data, surely that will add an immense CPU load to my NAS!"

                No. Most people have no idea how much time their NAS's CPU sits around waiting for data to be returned from the disk (hint: a lot). Modern CPUs are blisteringly fast. So fast that compressing 16kb of data and writing 2kb to disk is often 10x faster than just writing 16kb of data to disk in the first place.

                This is also why hardware RAID is dead - modern CPUs do this in their spare cycles, while they're waiting for other things to happen.

                scottalanmillerS tonyshowoffT 3 Replies Last reply Reply Quote -5
                • travisdh1T
                  travisdh1 @xrobau
                  last edited by travisdh1

                  @xrobau said in Changes at Sangoma:

                  So, as a Solaris administrator from way back, let's go through a couple of the misapprehensions about ZFS in that document you linked!

                  • ZFS Is an Alternative to RAID - Yes. It's a DIFFERENT TYPE of what normal people think of as 'RAID' - or specifically, RAID5/RAID6. They use Parity, and when a disk is broken/missing, it does calculations to figure out the missing data. ZFS uses copies of the data. Striping and Mirroring is obviously the same.

                  Uhm, what do you mean by "ZFS uses copies of the data" when talking about parity based arrays? This makes zero sense, as no matter what special sauce the array is using, it still must do the parity calculations. Unless it's really running a RAID 1/10 as another layer on arrays.

                  xrobauX 1 Reply Last reply Reply Quote 4
                  • xrobauX
                    xrobau @travisdh1
                    last edited by

                    @travisdh1 exactly! ZFS does not use parity. That seems to be where a bunch of this confusion is coming from.

                    Traditional RAID uses parity. ZFS uses copies.

                    travisdh1T scottalanmillerS 2 Replies Last reply Reply Quote -5
                    • travisdh1T
                      travisdh1 @xrobau
                      last edited by

                      @xrobau said in Changes at Sangoma:

                      • "ZFS Raid Levers aren't standard" - I have no idea what that means.

                      Neither do the people saying it, but we hear it said all the time.

                      xrobauX 1 Reply Last reply Reply Quote 0
                      • travisdh1T
                        travisdh1 @xrobau
                        last edited by

                        @xrobau said in Changes at Sangoma:

                        @travisdh1 exactly! ZFS does not use parity. That seems to be where a bunch of this confusion is coming from.

                        Traditional RAID uses parity. ZFS uses copies.

                        FFS, what are you talking about? ZFS can use copies of data, but that's exactly what RAID1/10 is. Almost every reference I've seen on ZFS are with people using parity levels with it. ZFS is not magic, it still must calculate parity.

                        What you're claiming is that ZFS can do nothing but mirrored drives

                        xrobauX 1 Reply Last reply Reply Quote 2
                        • xrobauX
                          xrobau @travisdh1
                          last edited by

                          @travisdh1 maybe they mean RAID LEVELS? Not levers? Which is correct. ZRAIDx means there are X ADDITIONAL copies of the data. ZRAID2 has 3 copies of each chunk, spread across physical devices. The "X" number is how many HDDs the zraid can tolerate failing.

                          The difference to RAID6 (for example) is that RAID6 has one copy of the data, but it's possible to figure out what the data was by looking at what remains and using the parity data to figure out what was on the missing disk(s)

                          travisdh1T scottalanmillerS 2 Replies Last reply Reply Quote -4
                          • xrobauX
                            xrobau @travisdh1
                            last edited by

                            @travisdh1 You still seem to be hung up on the RAID concept of parity. Truly, honestly, that's not how ZFS works. I cross my heart. It works on copies of data. Literal, exact, copies of the data.

                            Unfortunately, THIS is where the 'ZFS Can't expand a volume' problem comes into play. Because the location of those copies (on the physical spindles) is made by hashing the size and number of volumes in a zraid (I'm kinda simplifying here, but work with me!)

                            So if a ZRAID2 has 5 spindles, that means that a copy of block 1 of the zpool will be placed on spindle 1 at sector 10, 3 at sector 100 and spindle 5 at sector 500. The location of that doesn't need to be looked up, because it can be calculated.

                            The 'expansion' problem comes about because if you add another spindle to that ZRAID2, suddenly block 1 would be on spindle 1 at sector 10, spindle 3 at sector 500 and spindle 6 at sector 1000 (or whatever). So the location of the data would be wrong, and everything falls apart.

                            (Now, this is actually wrong, and there IS an index, but, that's advanced stuff, and this is good enough for you to get the idea).

                            Does that make more sense?

                            travisdh1T scottalanmillerS 3 Replies Last reply Reply Quote -5
                            • travisdh1T
                              travisdh1 @xrobau
                              last edited by

                              @xrobau said in Changes at Sangoma:

                              @travisdh1 maybe they mean RAID LEVELS? Not levers? Which is correct. ZRAIDx means there are X ADDITIONAL copies of the data. ZRAID2 has 3 copies of each chunk, spread across physical devices. The "X" number is how many HDDs the zraid can tolerate failing.

                              The difference to RAID6 (for example) is that RAID6 has one copy of the data, but it's possible to figure out what the data was by looking at what remains and using the parity data to figure out what was on the missing disk(s)

                              ZFS RAID levers aren't standard (they are very standard)

                              What is so difficult to understand about this statement?

                              xrobauX 1 Reply Last reply Reply Quote 1
                              • xrobauX
                                xrobau @travisdh1
                                last edited by

                                @travisdh1 I don't know what a RAID Lever is? Something to do with tuning?

                                travisdh1T 1 Reply Last reply Reply Quote -3
                                • travisdh1T
                                  travisdh1 @xrobau
                                  last edited by

                                  @xrobau said in Changes at Sangoma:

                                  @travisdh1 I don't know what a RAID Lever is? Something to do with tuning?

                                  Geesh, you're so bad as Scott.

                                  xrobauX 1 Reply Last reply Reply Quote 1
                                  • xrobauX
                                    xrobau @travisdh1
                                    last edited by

                                    @travisdh1 Dude, I've been doing this for almost 30 years, and I swear I have never heard of 'RAID levers'. RAID levels, sure. Not Levers. Can you explain what you mean?

                                    1 Reply Last reply Reply Quote -5
                                    • travisdh1T
                                      travisdh1 @xrobau
                                      last edited by

                                      @xrobau said in Changes at Sangoma:

                                      @travisdh1 You still seem to be hung up on the RAID concept of parity. Truly, honestly, that's not how ZFS works. I cross my heart. It works on copies of data. Literal, exact, copies of the data.

                                      Unfortunately, THIS is where the 'ZFS Can't expand a volume' problem comes into play. Because the location of those copies (on the physical spindles) is made by hashing the size and number of volumes in a zraid (I'm kinda simplifying here, but work with me!)

                                      So if a ZRAID2 has 5 spindles, that means that a copy of block 1 of the zpool will be placed on spindle 1 at sector 10, 3 at sector 100 and spindle 5 at sector 500. The location of that doesn't need to be looked up, because it can be calculated.

                                      The 'expansion' problem comes about because if you add another spindle to that ZRAID2, suddenly block 1 would be on spindle 1 at sector 10, spindle 3 at sector 500 and spindle 6 at sector 1000 (or whatever). So the location of the data would be wrong, and everything falls apart.

                                      (Now, this is actually wrong, and there IS an index, but, that's advanced stuff, and this is good enough for you to get the idea).

                                      Does that make more sense?

                                      Not at all, because you are describing RAID 5/6. There is nothing special or different here.

                                      xrobauX 1 Reply Last reply Reply Quote 5
                                      • xrobauX
                                        xrobau @travisdh1
                                        last edited by xrobau

                                        @travisdh1 Ahh, yeah, there is a difference though.

                                        RAID5 and 6 recover data based on parity.

                                        ZFS recovers data based on reading a copy of the data from another spindle.

                                        So, if a disk fails in a RAID5 (of, say 5 disks), the machine has to read the data from the remaining 4 disks, and then calculate what is missing by figuring it out from the parity data.

                                        If a disk fails in a ZRAID2 (of, again, 5 disks), it just reads the data from one of the other 2 copies (which it knows the location of already).

                                        This is where the zraid stuff scales. You can have a ZRAID8 for example (requiring 9 or more disks), where there are 8 copies of the data. Literal copies. The data is replicated onto those disks. There are no parity calculations, as there are with RAID5 and 6.

                                        So, as long as you can get the idea of parity out of your mind, suddenly the zraid stuff makes sense.

                                        Think of it like a traditional RAID10, but instead of the stripe and mirror being fixed in place, the data is load balanced across the physical spindles, and there is more than one mirror.

                                        scottalanmillerS 7 Replies Last reply Reply Quote -6
                                        • scottalanmillerS
                                          scottalanmiller @xrobau
                                          last edited by

                                          @xrobau said in Changes at Sangoma:

                                          RAID5 and 6 recover data based on parity.
                                          ZFS recovers data based on reading a copy of the data from another spindle.

                                          That's literally the same thing stated two different ways. The data "on the other spindle" is read via parity. ZFS is literally RAID 5 and 6 (and 7), it's not an alternative to them. If ZFS does it, those RAID levels do it.

                                          1 Reply Last reply Reply Quote 4
                                          • scottalanmillerS
                                            scottalanmiller @xrobau
                                            last edited by

                                            @xrobau said in Changes at Sangoma:

                                            So, if a disk fails in a RAID5 (of, say 5 disks), the machine has to read the data from the remaining 4 disks, and then calculate what is missing by figuring it out from the parity data.
                                            If a disk fails in a ZRAID2 (of, again, 5 disks), it just reads the data from one of the other 2 copies (which it knows the location of already).

                                            Yes, that's the same thing in both cases. The data on the disks in both examples is identical, what it can read from them is identical.

                                            1 Reply Last reply Reply Quote 3
                                            • 1
                                            • 2
                                            • 3
                                            • 4
                                            • 5
                                            • 6
                                            • 7
                                            • 1 / 7
                                            • First post
                                              Last post