Posts made by xrobau

OK, so you're obviously just trolling now. I've asked you nicely 4 times to stop spamming, and you're going out of your way to do it.

So fine, you win. I will not talk about ZFS here. You can say whatever you want, and I won't correct you. Enjoy.

@scottalanmiller Well, in this case because I'm trying to get across a REALLY COMPLEX thing, that you're having difficulty with, please respond IN ONE MESSAGE. That'll be easier to keep track of, OK? Otherwise there's no coherency.

@scottalanmiller No, because now you've just forked the conversation.

And look, you'll respond to this one, and then 15 things later you'll realise you never responded to the first thing.

@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:

Actually it is for you to make it easier to read, and easier to respond,

It does the exact opposite.

@scottalanmiller Dude, dumping a huge amount of stuff in 15 different posts is TOTALLY UNCOOL and is really unfriendly. Please don't do that. I now have to quote multiple things, scroll backwards and forwards, and generally waste even more of my time.

@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:

As the world moves to more reliable SSD, this has dropped off. It doesn't go away, but most corruption is believed to come from the network not the storage media, which nothing protects against (yet).

ZFS does. And, my experience is that I've had 2 SSDs silently fail and return corrupt data (of out 30 or so) and 2 spinning disks fail (out of several hundred). That's why I said it's a statistical anomaly.

@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:

and probably the average person will never see it in a life time.

They will probably never NOTICE it, unless they're running btrfs or ZFS, which has inherent checksum validation.

@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:

ZFS doesn't have that problem actually. You can grow a ZFS RAIDZ pool

No, you can't. In fact, the announcement of the POTENTIAL of the ability to do drew excitement from all the storage nerds. What you linked to is appending another zdev to a zpool. You can't expand a raidz.

https://www.reddit.com/r/homelab/comments/83wo88/any_news_on_zfs_raidz_expansion/

This is what frustrates me here - I know this stuff IN DEPTH (yes, I was wrong about parity vs copies - I dug up some of my old course notes and it said copies there - THAT was the source of my error), and you're trying to claim that you know this better than me, when you obviously don't. It's massively frustrating.

Re 'WTF are you doing with Hardware RAID, it's dead':
@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:

This is not true at all. Hardware RAID remains almost totally ubiquitous in commodity hardware

Funnily enough, almost all 'hardware RAID' cards are actually software RAID with a wrapper around them. And if they're not, they're going to be slower than your CPU anyway, so, back to my original point - why ADD slowness and INCREASE failure types? Pretty much the only Hardware RAID cards these days are the PERC-esque cards. I'm not going to go into EXPLICIT details, but if your RAID card doesn't have a heatsink on it, it's almost certainly a software raid implementation.

@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:

Few people promote software RAID as much as I do, but the benefits of hardware RAID are real and in no way is hardware RAID dead, dying, or useless.

Hardware RAID is slower, and more finnicky, and provides less visibility of individual platters than software RAID. For example, can a standard hardware RAID card provide access to SMART data of each drive? (No).

@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:

I've done hundreds or thousands of hardware card swaps and the number of failed imports was zero.

As I said originally - the only way that is true is if you had identical cards with identical firmware versions on standby. That's perfectly fine for an EMC sized company, but it's not fine for anyone with only 200 or 300 spindles. I've had multiple P410i's refuse to import a RAID that was generated with a different version of firmware. This is not something uncommon, this is something that happens ALL THE TIME.

@scottalanmiller said in Revisiting ZFS and FreeNAS in 2019:

. You picked up the challenge and decided to take the list of concerns and attempt to refute many or most of them to show why you felt ZFS (and FreeNAS) were good choices.

FreeNAS is just a wrapper for ZFS, with all the tools everyone needs built in.

ZFS is, unfortunately for those that are trying to make a living in the HARDWARE RAID space, a significant nail in their coffin. I brought up a whole bunch of things where your statements were wrong, or misleading, or in some cases totally irrelevant.

In retrospect, from your comments, it seems that that you make a living from hardware RAID, so it's somewhat unsurprising that you're trying to spread a pile of FUD on ZFS. Comments like 'people say it's magic' are just casting dispersion on it, purely to disparage it without that meaning anything.

And ZFS is so portable that I can literally pull the drives from a FreeNAS box, plug them into an Ubuntu machine, run 'zfs import' and all my data is there. Can you do that when you move your HDDs from a HP to a Dell to an IBM?

There. See how you can reply in ONE comment, rather than 30? It makes it much more constructive.

Sigh.

@scottalanmiller said in Changes at Sangoma:

You started with some mildly incorrect beliefs based on common marketing.

I'm a sysadmin. I've done courses. This is not marketing, this is training. Now, maybe I've been trained wrong, or maybe I've forgotten my training - I was actually looking through the ZoL source code to see if I was wrong (and, it looks like I may be!), but I'm just not interested any more.

You're so hung up about HOW BLOCKS ARE STORED ON THE DISK that you've ignored everything else I've said.

So. Whatever. I'm REALLY done this time.

Oh, and here's the link to the code where they ARE doing math, which is when I was about to come in and go 'Whoops, looks like I was wrong', but turns out you're just interested in being an arsehole, rather than actually engaging in discussion.

https://github.com/zfsonlinux/zfs/blob/a8577bdb32e091645df901d8501e44ef50748389/module/zfs/vdev_raidz_math_impl.h#L525

@scottalanmiller

@scottalanmiller said in Changes at Sangoma:

RAID 2 (no existing implementation)
RAID 3 (deprecated)
RAID 4 (rare)
RAID 5 (aka RAIDZ)
RAID 6 (aka RAIDZ2)
RAID 7 (aka RAIDZ3)

No. RAIDZ does not use parity. Just because people REFER to it as parity does not mean it is such. That is the 'simplification confusion'.

So yeah, sorry. There's no use continuing this discussion because you're adamant that you're correct, and you refuse to even think you could be wrong. So, being that there is no possible way for me to change you mind, there's no use me continuing, is there?

OK, here's my last ditch gasp to try to get you to understand my point of view:

RAID1 doesn't use parity. RAID0 doesn't use parity. So, which RAID versions do use parity, being that this is what this entire discussion is about?

@scottalanmiller Sorry, dude, I'm going to give up. If you don't want to work with me here, then I'm just going to not bother.

I have no coin in this game. I'm just trying to help you out. I've been using ZFS for 15 years now, and I'm extremely confident in my knowledge. A lot of people try to simplify this and those simplifications are where you're getting confused.

Anyway, I'm out. Enjoy!

I'm not going to bother going through all the individual replies - please try to consolidate them into a single response, but almost all of them are suffering from the same misapprehension that ZFS uses parity data instead of copies. If I missed something (I skimmed through them), feel free to reply in a single post and I'll try to address any confusion.

@scottalanmiller said in Changes at Sangoma:

@xrobau dont' get us wrong, we totally know that you are a developer, not IT.

Funnily enough, you're wrong there - I'm a developer NOW, but I'm a Solaris Sysadmin originally. Then I got my Windows cert, and CCIE, and a bunch of other things before moving into DevOps. So, please - trust me when I say I know what I'm talking about.

@scottalanmiller said in Changes at Sangoma:

These are not copies, these are "pieces of the block". Each spindle gets one piece of it and you need at least three of the five spindles to put the data back together

Honestly, this is where you are 100% wrong, and you refuse to listen to me. I'm trying to explain how ZFS is different, and you can't just say 'You're wrong, and I know this because I know nothing about ZFS'.

ZFS is based on copies of the data. There is no parity. Stop using the word parity as it has NOTHING to do with ZFS. If you're using the word parity, in relation to ZFS, you are wrong.

I don't know how much more blunt I can be. ZFS does not use Parity. ZFS uses copies.

Right, now that I hopefully have made that clear, let me try again.

Parity, in RAID speak is 1+2+3+4=10 - If you lose one of the disks, you end up with this:

1+?+3+4=10

Simple maths lets you figure out that the missing value is 2. (10-4-3-1 = 2)

That's how parity works. Not rocket science.

ZFS works on copies. So, when you write 1, 2, 3 and 4 to a zpool, you get something like this:

Disk 1: x 1 x 3 x
Disk 2: 1 x 2 x 3
Disk 3: x 1 2 x 4
Disk 4: 4 x 2 x 3
Disk 5: x x x x 4

Copies. Of. The. Data.

That looks vaguely accurate, but even if I missed something, assume 3 copies of all data across 5 spindles.

Copies. Not parity. COPIES.

OK, so can we move on from this now? Old RAID == Parity. ZFS == Copies. Hopefully I've made this clear now.

Now, if you want to learn more about this, please feel free to go on any of the Solaris Administration courses I have, OR, feel free to read any of the plethora of documentation on ZFS. But telling me I'm wrong isn't going to get you anywhere, because I know what I'm talking about here. This is my field of expertise.

Now, if you can take a breath, admit that you've learned something new about ZFS, I can continue on with the OTHER differences, and some of your potential misconceptions

@travisdh1 Ahh, yeah, there is a difference though.

RAID5 and 6 recover data based on parity.

ZFS recovers data based on reading a copy of the data from another spindle.

So, if a disk fails in a RAID5 (of, say 5 disks), the machine has to read the data from the remaining 4 disks, and then calculate what is missing by figuring it out from the parity data.

If a disk fails in a ZRAID2 (of, again, 5 disks), it just reads the data from one of the other 2 copies (which it knows the location of already).

This is where the zraid stuff scales. You can have a ZRAID8 for example (requiring 9 or more disks), where there are 8 copies of the data. Literal copies. The data is replicated onto those disks. There are no parity calculations, as there are with RAID5 and 6.

So, as long as you can get the idea of parity out of your mind, suddenly the zraid stuff makes sense.

Think of it like a traditional RAID10, but instead of the stripe and mirror being fixed in place, the data is load balanced across the physical spindles, and there is more than one mirror.

@travisdh1 Dude, I've been doing this for almost 30 years, and I swear I have never heard of 'RAID levers'. RAID levels, sure. Not Levers. Can you explain what you mean?

@travisdh1 I don't know what a RAID Lever is? Something to do with tuning?

@travisdh1 You still seem to be hung up on the RAID concept of parity. Truly, honestly, that's not how ZFS works. I cross my heart. It works on copies of data. Literal, exact, copies of the data.

Unfortunately, THIS is where the 'ZFS Can't expand a volume' problem comes into play. Because the location of those copies (on the physical spindles) is made by hashing the size and number of volumes in a zraid (I'm kinda simplifying here, but work with me!)

So if a ZRAID2 has 5 spindles, that means that a copy of block 1 of the zpool will be placed on spindle 1 at sector 10, 3 at sector 100 and spindle 5 at sector 500. The location of that doesn't need to be looked up, because it can be calculated.

The 'expansion' problem comes about because if you add another spindle to that ZRAID2, suddenly block 1 would be on spindle 1 at sector 10, spindle 3 at sector 500 and spindle 6 at sector 1000 (or whatever). So the location of the data would be wrong, and everything falls apart.

(Now, this is actually wrong, and there IS an index, but, that's advanced stuff, and this is good enough for you to get the idea).

Does that make more sense?

@travisdh1 maybe they mean RAID LEVELS? Not levers? Which is correct. ZRAIDx means there are X ADDITIONAL copies of the data. ZRAID2 has 3 copies of each chunk, spread across physical devices. The "X" number is how many HDDs the zraid can tolerate failing.

The difference to RAID6 (for example) is that RAID6 has one copy of the data, but it's possible to figure out what the data was by looking at what remains and using the parity data to figure out what was on the missing disk(s)

@travisdh1 exactly! ZFS does not use parity. That seems to be where a bunch of this confusion is coming from.

Traditional RAID uses parity. ZFS uses copies.

@scottalanmiller Oh, and I forgot one of the most important things - live compression and decompression.

"What?" I hear you say, "why would I want to compress and decompress my data, surely that will add an immense CPU load to my NAS!"

No. Most people have no idea how much time their NAS's CPU sits around waiting for data to be returned from the disk (hint: a lot). Modern CPUs are blisteringly fast. So fast that compressing 16kb of data and writing 2kb to disk is often 10x faster than just writing 16kb of data to disk in the first place.

This is also why hardware RAID is dead - modern CPUs do this in their spare cycles, while they're waiting for other things to happen.

Well wow that did kinda go off on a bit of a tangent. As a bit of a rebuttal, you're kinda getting confused and claming FreeNAS is a bunch of things it isn't. In fact, in that 'common myths' page is ... I dunno, random made up stuff?

So let's take it from MY perspective - Basically, FreeNAS is a really snazzy front end to ZFS. That's it. It ALREADY comes with a bunch of scripts to monitor SMART, resilver, validate data integrity, and send me a warm-and-fuzzy email every week saying that everything is wonderful. That's pretty much it.

So, Why is ZFS so cool? Because it has MULTIPLE checks on data integrity. In fact, the whole design of ZFS was based around integrity, not speed. If you want fast, you don't use ZFS (or, you throw a bunch of CPU and RAM and Disks and stuff at it).

Data Integrity is BY FAR the most important thing in data storage today. If you have a HDD with bad cache ram on it, ZFS (and, to a lesser extent, btrfs) is the only thing that's going to warn you that something is wrong with that device (and it's not like this is uncommon - I've discovered two drives in the last year that were writing corrupt data). This happens.

So, as a Solaris administrator from way back, let's go through a couple of the misapprehensions about ZFS in that document you linked!

• ZFS Is an Alternative to RAID - Yes. It's a DIFFERENT TYPE of what normal people think of as 'RAID' - or specifically, RAID5/RAID6. They use Parity, and when a disk is broken/missing, it does calculations to figure out the missing data. ZFS uses copies of the data. Striping and Mirroring is obviously the same.
• "ZFS Raid Levers aren't standard" - I have no idea what that means.
• ZFS Does not work on hardware RAID - It DOES work. You just SHOULDN'T use it with Hardware RAID. In fact, Hardware RAID is a terrible idea, and has been so for many many years now. Until hardware RAID manufacturers start putting 3ghz Xeons on their RAID cards, software raid is always going to be faster (and - more importantly - much easier to recover from a hardware failure). But the design of ZFS is that it talks to the raw spinning disks. Adding other abstraction layers in the middle is counterproductive.
• That bit rot affects people every day and only ZFS can protect you - No, btrfs can protect you, too. Apart from that, yes - it does. As I said above, twice in about 18 months for me, and I only have about 100 spinning disks. I'm not going to get into SSDs, because I've had terrible results with a couple that wildly skew the stats - but they've burned me so badly, that the only way I'm trusting SSDs these days is in a ZRAID2 (eg, 3 copies of the data on 3 devices).
• ... Hardware RAID is useless - Yep, these days it's a terrible idea to use hardware RAID, unless you're prepared to keep identical hardware cards with identical firmware versions in stock, right next to every machine with hardware RAID. If you haven't been bitten by a hardware RAID card being unable to import a RAID when the previous card died, you're a lucky lucky person. Also, it's slower! Why do you want to ADD unreliability, and REMOVE speed? Hardware RAID is dead. It died many years ago
• "XXX is special" - Don't know why you think someone would say that. So let's just skip over that.
• Scrubbing is special - Yes it is! If you have a disk that is faulty, scrubbing will pick it up and repair the damage. That can only happen with ZFS, because it has COPIES of the data, and every copy has a checksum. If one of the copies doesn't match the checksum, it'll repair it by writing over it with one of the good copies.

The other two points aren't really relevant, and don't know where they're coming from.

So, after that nerd braindump, I am your guest ZFS expert! Ask me anything

@JaredBusch don't forget I left at the end of last year, and I'm now working with Tony at his new business Clearly IP.

One of the awesome things about GPL code is what I tweeted recently - https://twitter.com/xrobau/status/1126587963242999809 - if it's GPL, it's yours forever, basically.

I can also happily say that Clearly IP is not doing anything with FreePBX (at the moment, anyway) - I will personally still be looking after the Firewall module, on phonebo.cx, but that's about the limit of our involvement.

We have a bunch of cool ideas planned, and if you're friends with me on facebook (I'm xrobau there, too - of course) it won't take much brain sweat to figure out what that is, even though it's TECHNICALLY confidential.

There are also other people that are leaving/left - I know that David Duffet and Matt Jordan are public knowledge. I'm not sure how many of the other resignations have been made public yet, so I'm not going to say anything about them