Concerns with BtrFS and ReFS
-
Remember, in storage, .01% data loss is too high to even think about calling production. Terms like "norm" have no place, because anything in the 51% range is too low to be usable statistically. All storage items need to be reliable to a point where concepts like "norm" are meaningless. This is what gets people with RAID. A business owner wants six nines of durability, but IT people will often point out that at least 51% of people don't lose data and treat that as similar to 99.9999% but they are mathematically ridiculously far apart.
-
Look at the kinds of issues Veeam currently sees with ReFS. These are not the kinds of issues one expects from a mature, reliable filesystem. Third party software can cause issues, but having to avoid AV because the FS can't handle it is pretty flaky behaviour.
-
@scottalanmiller said in KVM and Back Ups:
Look at the kinds of issues Veeam currently sees with ReFS. These are not the kinds of issues one expects from a mature, reliable filesystem. Third party software can cause issues, but having to avoid AV because the FS can't handle it is pretty flaky behaviour.
You mean disabling AV isn't a standard practice that everyone should employ?!
-
@dafyre said in KVM and Back Ups:
@scottalanmiller said in KVM and Back Ups:
@dafyre said in KVM and Back Ups:
In my experience with it, it has often corrupted randomly and to the point that it's own snapshots are no help, nor are VMware Snapshots.
How could it correct VMware snapshots?
I guess it's more that BtrFS doesn't detect the corruption early enough and our VMware snapshot are nothing but snapshots of corrupt data... That's about the only way I can explain it.
When you recovered and did investigation, you determined that the filesystem, not the data on the filesystem, was corrupted? No filesystem can detect the latter. How did you figure out that BtrFS was to "blame", and what did you move to to address the issue? Only ZFS would even offer an alternative.
-
@scottalanmiller said in KVM and Back Ups:
@KOOLER on ReFS performance issues... https://www.starwindsoftware.com/blog/log-structured-file-systems-microsoft-refs-v2-investigation-part-1
Wow they said Engineers set that up? Obviously not the IT type of Engineer. The whole thing is totally wrong, totally and completely unsupported in just about every way, and in no way supportive of what you tried to prove with it considering the given setup... Ä
-
@Obsolesce said in Concerns with BtrFS and ReFS:
@scottalanmiller said in KVM and Back Ups:
@KOOLER on ReFS performance issues... https://www.starwindsoftware.com/blog/log-structured-file-systems-microsoft-refs-v2-investigation-part-1
Wow they said Engineers set that up? Obviously not the IT type of Engineer. The whole thing is totally wrong, totally and completely unsupported in just about every way, and in no way supportive of what you tried to prove with it considering the given setup... Ä
How much special consideration does ReFS need to work? Obviously it can't require Windows Software RAID or it wouldn't be production ready in the slightest. What all special knowledge must people have to use a filesystem? And why does ReFS need to much but not NTFS? Just needing lots of special knowledge to use a FS seems like admission that it has a lot of problems.
-
@Obsolesce said in Concerns with BtrFS and ReFS:
Wow they said Engineers set that up? Obviously not the IT type of Engineer.
MS Storage / Kernel MVP.
-
We use EXT4 in our linux storage appliances - yet they seem to be pushing btrfs?
-
@MattSpeller said in Concerns with BtrFS and ReFS:
We use EXT4 in our linux storage appliances - yet they seem to be pushing btrfs?
Synology and ReadyNAS seem to push BtrFS. It makes things easier for them.
For production, everyone I know pushes XFS. Fast and reliable. Pretty much the only big factors in storage.
-
@scottalanmiller said in Concerns with BtrFS and ReFS:
For production, everyone I know pushes XFS. Fast and reliable. Pretty much the only big factors in storage.
Same
-
@scottalanmiller said in Concerns with BtrFS and ReFS:
@MattSpeller said in Concerns with BtrFS and ReFS:
We use EXT4 in our linux storage appliances - yet they seem to be pushing btrfs?
Synology and ReadyNAS seem to push BtrFS. It makes things easier for them.
For production, everyone I know pushes XFS. Fast and reliable. Pretty much the only big factors in storage.
More reliable than EXT4? We are going to replace our primary file storage NAS's and I just want something reliable that won't give me any (swear) headaches later
-
@MattSpeller said in Concerns with BtrFS and ReFS:
More reliable than EXT4?
Absolutely, specifically more reliable and faster (for most workloads) than EXT4. EXT4 is considered a desktop FS, while XFS is the server FS. EXT4 is tuned for the flexibility and small file sizes of a desktop. XFS for the performance, reliability, and planning of a server.
-
@scottalanmiller said in Concerns with BtrFS and ReFS:
@MattSpeller said in Concerns with BtrFS and ReFS:
More reliable than EXT4?
Absolutely, specifically more reliable and faster (for most workloads) than EXT4. EXT4 is considered a desktop FS, while XFS is the server FS. EXT4 is tuned for the flexibility and small file sizes of a desktop. XFS for the performance, reliability, and planning of a server.
Awesome - would you still suggest it if the storage was for ~4TB of 1mb documents accessed live and regularly and ~8TB+ of video?
-
@scottalanmiller said in Concerns with BtrFS and ReFS:
@MattSpeller said in Concerns with BtrFS and ReFS:
More reliable than EXT4?
Absolutely, specifically more reliable and faster (for most workloads) than EXT4. EXT4 is considered a desktop FS, while XFS is the server FS. EXT4 is tuned for the flexibility and small file sizes of a desktop. XFS for the performance, reliability, and planning of a server.
If your only choices were EXT4 or BtrFS - ext4 every time?
-
@MattSpeller said in Concerns with BtrFS and ReFS:
@scottalanmiller said in Concerns with BtrFS and ReFS:
@MattSpeller said in Concerns with BtrFS and ReFS:
More reliable than EXT4?
Absolutely, specifically more reliable and faster (for most workloads) than EXT4. EXT4 is considered a desktop FS, while XFS is the server FS. EXT4 is tuned for the flexibility and small file sizes of a desktop. XFS for the performance, reliability, and planning of a server.
If your only choices were EXT4 or BtrFS - ext4 every time?
Not every time, but generally. I'm not a fan of the ZFS and related filesystems in general (BtrFS, ReFS, etc.) They are full of gimics and rely on RAID integration for most of their touted features (and mostly it's the RAID, not the FS, doing the work - very misleading.) It's not that the ideas are all bad, but it makes the FS way too complex and confusing causing generally more issues than it solves. Also, XFS, EXT*, NTFS are built for speed and normal usage. ZFS and similar are build for resilience under specific use cases, massive storage, virtualization storage (nested filesystems) and stuff like that - niche case.
-
@scottalanmiller said in Concerns with BtrFS and ReFS:
@MattSpeller said in Concerns with BtrFS and ReFS:
We use EXT4 in our linux storage appliances - yet they seem to be pushing btrfs?
Synology and ReadyNAS seem to push BtrFS. It makes things easier for them.
For production, everyone I know pushes XFS. Fast and reliable. Pretty much the only big factors in storage.
XFS also has
xfsdumpandxfsrestore. It's not exactly the same as btrfs but it does give some backup ability. It also has dedupe but it's somewhat limited last I saw https://hooks.technology/2018/03/xfs-deduplication-with-reflinks/ -
@dafyre said in Concerns with BtrFS and ReFS:
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
For example, I had a developer fubar a server the other day. Completely unrecoverable. It was hosted at vultr, and I used their backup service. I was able to completely restore the server from their snapshot backup. That’s what I am after.
That's not crash consistent. So THAT level of backup KVM can do without anything special, it's just taking a snapshot of the storage. You have that with any system because it is done at the storage layer.
What tools can I use to do that (scheduled) with KVM on fedora?
If you want the Vultr style (or ProxMox risky style), you can do that right from the storage layer. So first determine the storage that you are going to use. ZFS, BtrFS, XFS, LVM, etc. Then you use the native tools (if you want) to snap it. Everything except the scheduling is just built in.
What is the latest recommendation for storage now? LVM?
LVM, ZFS, BtrFS are all fine. I've not used this but here is a script to do LVM backups...
I am going on record here as recommending that you stay away from BtrFS... far, far away. I can't say anything about ZFS as I haven't used that... But around my office, we avoid BtrFS like the plague.
I don't have any real world experience with data loss but I've read a lot of cases where people have had it. I think a lot of that might stem from people trying to use RAID 5/6 with it and that is somehow still unstable. I'm guessing that's why Red Hat dropped it and decided to make Stratis.
-
@stacksofplates said in Concerns with BtrFS and ReFS:
@dafyre said in Concerns with BtrFS and ReFS:
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
@scottalanmiller said in KVM and Back Ups:
@fuznutz04 said in KVM and Back Ups:
For example, I had a developer fubar a server the other day. Completely unrecoverable. It was hosted at vultr, and I used their backup service. I was able to completely restore the server from their snapshot backup. That’s what I am after.
That's not crash consistent. So THAT level of backup KVM can do without anything special, it's just taking a snapshot of the storage. You have that with any system because it is done at the storage layer.
What tools can I use to do that (scheduled) with KVM on fedora?
If you want the Vultr style (or ProxMox risky style), you can do that right from the storage layer. So first determine the storage that you are going to use. ZFS, BtrFS, XFS, LVM, etc. Then you use the native tools (if you want) to snap it. Everything except the scheduling is just built in.
What is the latest recommendation for storage now? LVM?
LVM, ZFS, BtrFS are all fine. I've not used this but here is a script to do LVM backups...
I am going on record here as recommending that you stay away from BtrFS... far, far away. I can't say anything about ZFS as I haven't used that... But around my office, we avoid BtrFS like the plague.
I don't have any real world experience with data loss but I've read a lot of cases where people have had it. I think a lot of that might stem from people trying to use RAID 5/6 with it and that is somehow still unstable. I'm guessing that's why Red Hat dropped it and decided to make Stratis.
yeah, there is a strong tendency with these kinds of filesystems to see them as "magic" and try to do things in ways you would never do with a different kind of FS, even though the risks are the same.
-
Had a customer this week have BtrFS cause all kinds of performance issues. Made a NAS unusable. Switched to ext4, everything was fine.
-
BtrFS is what Synology uses?
