Linux: BtrFS
-
BtrFS, the "Better File System" pronounced "Butter F S", is a Linux-native, very modern addition to the Linux ecosystem meant primarily to provide Linux with a solid competitive product to FreeBSD's ZFS (BtrFS was introduced before ZFS was widely available on Linux.) BtrFS was first introduced by Oracle, who also makes ZFS. Both of these are classified as Copy-on-Write (COW) filesystems.
Before we delve into BtrFS, we need to clarify what exactly it is. Like the elder ZFS, BtrFS is far more than a filesystem. It is actually three very distinct components:
- File System: BtrFS contains a full POSIX filesystem and allows it to operate anywhere that EXT3, EXT4, XFS or ZFS would be used, for example. It is not a clustered filesystem, which is a common misconception. It is a traditional filesystem.
- Logical Volume Manager: BtrFS, like ZFS, contains a complete logical volume manager (LVM) within itself and has no need to rely on Linux' LVM2 for this functionality but, like any filesystem or LVM, could be used on top of another LVM like LVM2 if desired.
- Software RAID: Also like ZFS, BtrFS contains its own software RAID implementation so can function in a software RAID capacity without needing to use the older MD RAID system.
With these three components BtrFS, like ZFS, is really a complete storage subsystem rather than strictly a filesystem. BtrFS is a "stack" of RAID / LVM / FS and would be more comparable to another stack such as MD / LVM2 / XFS than to any single component of another stack. This adds complication as discussing BtrFS requires that we understand which layers are being considered at any give time. For example we could easily use these stacks at different times: MD / LVM2 / BtrFS; or MD / BtrFS / BtrFS; or BtrFS / BtrFS / BtrFS!
BtrFS is considered the "path forward" by the EXT team and was started by an engineer from the Reiser team. By 2017 with XFS having widely displaced the EXT family and Reiser family of filesystems the future of Linux is primarily looking to be shared by XFS and BtrFS. In 2017, at the time of this writing, Ubuntu is still on EXT4 as the default filesystem, Red Hat has moved to XFS and Suse has moved to BtrFS as defaults.
BtrFS is an extremely fast moving and elusive target to describe as development is ongoing and rapid and many core features are planned but not yet implemented. At this time the following are the key features of the BtrFS storage stack:
- Mostly self-healing in some configurations due to the nature of copy-on-write
- Online defragmentation and an autodefrag mount option
- Online volume growth and shrinking
- Online block device addition and removal
- Online balancing (movement of objects between block devices to balance load)
- Offline filesystem check
- Online data scrubbing for finding errors and automatically fixing them for files with redundant copies
- RAID 0, RAID 1, and RAID 10
- Subvolumes (one or more separately mountable filesystem roots within each disk partition)
- Transparent compression (zlib and LZO), configurable per file or volume
- Snapshots (read-only or copy-on-write clones of subvolumes)
- File cloning (copy-on-write on individual files, or byte ranges thereof)
- Checksums on data and metadata
- Union mounting of read-only storage, known as file system seeding (read-only storage used as a copy-on-write backing for a writable Btrfs)
- Block discard (reclaims space on some virtualized setups and improves wear leveling on SSDs with TRIM)
- Send/receive (saving diffs between snapshots to a binary stream)
- Hierarchical per-subvolume quotas
- Out-of-band data deduplication (requires userspace tools)
A great number of advanced features are planned, but are not yet available. Key among these are parity RAID features including RAID 5 and 6, but also potentially the first implementations of RAID 5.4, 5.5 and 5.6 and the only other RAID 5.3 (aka RAID 7) implementation outside of ZFS. Object based mirrored RAID, inline deduplication and encryption are also planned for the near future.
Already BtrFS is an advanced and powerful choice for Linux admins when suitable. Many of its features are designed for use when used as the physical layer storage system and not for use in a VM. For most purposes, we will expect to find BtrFS used in dedicated storage devices and other choices more commonly used in virtual machine instances.
Competition in other operating systems: Microsoft's answer to ZFS and BtrFS is their ReFS COW filesystem. Mac OSX abandoned ZFS in favour of their upcoming APFS COW system. Dragonfly has its own advanced COW filesystem named Hammer.
Part of a series on Linux Systems Administration by Scott Alan Miller
-
@scottalanmiller said in Linux: BtrFS:
vanced features are planned, but are not yet available. Key among these are parity RAID features including RAID 5 and 6, but also potentially the first implementations of RAID 5.4, 5.5 and 5.6 and the only other RAID 5.3 (aka RAID 7) implementation outside of ZFS. Object based mirrored RAID, inline deduplication and encryption are also planned for the near future.
I hear alot lately more about those filesystems, but as far as I recall ZFS requires alot of RAM as it the same with BTRFS? and is this only when you need to configure RAID with it?
I head alot of people like them for the ability to snapshot, and self healing, I understood the self healing part, but for snapshots, imagine I will run KVM server ontop of BTRFS, will it makes sense to snapshot the volume ?
or just the VM, which is better ?And yes I use Centos latest, I noticed they have switched to XFS even in the boot partition, does it support volume snapshots?
Thanks
-
@msff-amman-Itofficer said in Linux: BtrFS:
I hear alot lately more about those filesystems, but as far as I recall ZFS requires alot of RAM as it the same with BTRFS?
No, and no. The need for a lot of RAM is specifically around ZFS if deduplication is turned on. Then it is 1GB per 1TB. Unless you are using that feature, ZFS has no noticeably increased RAM needs.
-
@msff-amman-Itofficer said in Linux: BtrFS:
and is this only when you need to configure RAID with it?
RAID uses very little RAM. Cache, would use a lot of RAM, though. But that is configurable.
-
@msff-amman-Itofficer said in Linux: BtrFS:
I head alot of people like them for the ability to snapshot,
This is one of the silliest things that people talk about. LVM2 has offered snapshots on Linux for two decades. We don't need BtrFS or ZFS for that functionality. It's very common to snapshot on Linux even without these filesystems. It's only because they both have the LVM layer built in that they have this. All filesystems can be snapshot.
-
@msff-amman-Itofficer said in Linux: BtrFS:
... imagine I will run KVM server ontop of BTRFS, will it makes sense to snapshot the volume ?
Did it make sense before with LVM and XFS? No, and the reason is because you can't snap outside of the VM because there isn't enough information to know if the data is quiesced. Snapping under the hood at the LVM2 or BtrFS layer would be the same as snapping on a SAN which, of course, can cause corruption. So we avoid that. We need the VM itself to have this done via the hypervisor.
-
@msff-amman-Itofficer said in Linux: BtrFS:
And yes I use Centos latest, I noticed they have switched to XFS even in the boot partition, does it support volume snapshots?
Yes, and no. Filesystems don't do snapshots, logical volume managers do. XFS used properly on LVM2 has always had snapshots. Just like EXT2, 3 & 4 have always supported. Or JFS. The filesystem can't block the snapping functionality, so no matter what you put on top of an LVM, even if it isn't a filesystem, can be snapped as the filesystem doesn't not need to grant permission for this operation.
-
If it's "Better File System", why not pronounce it better, rather than butter?
-
@BBigford said in Linux: BtrFS:
If it's "Better File System", why not pronounce it better, rather than butter?
I've never understood that either, and a few people do, but somehow "butter" became the de facto standard, even before it was available for general use. My only guess is that they did not want to come off as pretentious.
-
@scottalanmiller said in Linux: BtrFS:
@msff-amman-Itofficer said in Linux: BtrFS:
... imagine I will run KVM server ontop of BTRFS, will it makes sense to snapshot the volume ?
Did it make sense before with LVM and XFS? No, and the reason is because you can't snap outside of the VM because there isn't enough information to know if the data is quiesced. Snapping under the hood at the LVM2 or BtrFS layer would be the same as snapping on a SAN which, of course, can cause corruption. So we avoid that. We need the VM itself to have this done via the hypervisor.
I've specifically seen you recommend using LVM as a backup mechanism with KVM.
-
@stacksofplates said in Linux: BtrFS:
@scottalanmiller said in Linux: BtrFS:
@msff-amman-Itofficer said in Linux: BtrFS:
... imagine I will run KVM server ontop of BTRFS, will it makes sense to snapshot the volume ?
Did it make sense before with LVM and XFS? No, and the reason is because you can't snap outside of the VM because there isn't enough information to know if the data is quiesced. Snapping under the hood at the LVM2 or BtrFS layer would be the same as snapping on a SAN which, of course, can cause corruption. So we avoid that. We need the VM itself to have this done via the hypervisor.
I've specifically seen you recommend using LVM as a backup mechanism with KVM.
Have I recommended it, or have I recommended it within the context of taking image based backups?
-
@scottalanmiller said in Linux: BtrFS:
@stacksofplates said in Linux: BtrFS:
@scottalanmiller said in Linux: BtrFS:
@msff-amman-Itofficer said in Linux: BtrFS:
... imagine I will run KVM server ontop of BTRFS, will it makes sense to snapshot the volume ?
Did it make sense before with LVM and XFS? No, and the reason is because you can't snap outside of the VM because there isn't enough information to know if the data is quiesced. Snapping under the hood at the LVM2 or BtrFS layer would be the same as snapping on a SAN which, of course, can cause corruption. So we avoid that. We need the VM itself to have this done via the hypervisor.
I've specifically seen you recommend using LVM as a backup mechanism with KVM.
Have I recommended it, or have I recommended it within the context of taking image based backups?
That's what he was talking about above. Using BtrFS to take a snapshot of the volume, like using LVM to take a snapshot of the volume.
-
@stacksofplates said in Linux: BtrFS:
@scottalanmiller said in Linux: BtrFS:
@stacksofplates said in Linux: BtrFS:
@scottalanmiller said in Linux: BtrFS:
@msff-amman-Itofficer said in Linux: BtrFS:
... imagine I will run KVM server ontop of BTRFS, will it makes sense to snapshot the volume ?
Did it make sense before with LVM and XFS? No, and the reason is because you can't snap outside of the VM because there isn't enough information to know if the data is quiesced. Snapping under the hood at the LVM2 or BtrFS layer would be the same as snapping on a SAN which, of course, can cause corruption. So we avoid that. We need the VM itself to have this done via the hypervisor.
I've specifically seen you recommend using LVM as a backup mechanism with KVM.
Have I recommended it, or have I recommended it within the context of taking image based backups?
That's what he was talking about above. Using BtrFS to take a snapshot of the volume, like using LVM to take a snapshot of the volume.
Well yes, but what I mean is if someone asks "how do you take backups" I normally recommend an agent. If someone says "how do I snap images" I normally say LVM.
The difference is if the goal is to backup the system or if the question is "can KVM do image backups."
-
@scottalanmiller said in Linux: BtrFS:
@BBigford said in Linux: BtrFS:
If it's "Better File System", why not pronounce it better, rather than butter?
I've never understood that either, and a few people do, but somehow "butter" became the de facto standard, even before it was available for general use. My only guess is that they did not want to come off as pretentious.
Lol then pronounce it 'butter' and call it 'Butter File System'?
-
@BBigford said in Linux: BtrFS:
@scottalanmiller said in Linux: BtrFS:
@BBigford said in Linux: BtrFS:
If it's "Better File System", why not pronounce it better, rather than butter?
I've never understood that either, and a few people do, but somehow "butter" became the de facto standard, even before it was available for general use. My only guess is that they did not want to come off as pretentious.
Lol then pronounce it 'butter' and call it 'Butter File System'?
It is what it is, I just report the news
-
@scottalanmiller said in Linux: BtrFS:
@stacksofplates said in Linux: BtrFS:
@scottalanmiller said in Linux: BtrFS:
@stacksofplates said in Linux: BtrFS:
@scottalanmiller said in Linux: BtrFS:
@msff-amman-Itofficer said in Linux: BtrFS:
... imagine I will run KVM server ontop of BTRFS, will it makes sense to snapshot the volume ?
Did it make sense before with LVM and XFS? No, and the reason is because you can't snap outside of the VM because there isn't enough information to know if the data is quiesced. Snapping under the hood at the LVM2 or BtrFS layer would be the same as snapping on a SAN which, of course, can cause corruption. So we avoid that. We need the VM itself to have this done via the hypervisor.
I've specifically seen you recommend using LVM as a backup mechanism with KVM.
Have I recommended it, or have I recommended it within the context of taking image based backups?
That's what he was talking about above. Using BtrFS to take a snapshot of the volume, like using LVM to take a snapshot of the volume.
Well yes, but what I mean is if someone asks "how do you take backups" I normally recommend an agent. If someone says "how do I snap images" I normally say LVM.
The difference is if the goal is to backup the system or if the question is "can KVM do image backups."
I don't understand what this has to do with anything. Above you said
Did it make sense before with LVM and XFS? No, and the reason is because you can't snap outside of the VM because there isn't enough information to know if the data is quiesced.
It does make sense. You can easily suspend the VM and take a LV snapshot.
You've also said this to someone else which is what I was referring to:
Why are you not using logical volumes? that is both a general Linux best practice as well as the backup method for KVM.
Then you linked a python script that does exactly what I said. Suspends the VM and takes a snapshot of the logical volume.
My point is, why are you saying here that it makes no sense?
-
@stacksofplates said in Linux: BtrFS:
Did it make sense before with LVM and XFS? No, and the reason is because you can't snap outside of the VM because there isn't enough information to know if the data is quiesced.
It does make sense. You can easily suspend the VM and take a LV snapshot.
You've also said this to someone else which is what I was referring to:
Why are you not using logical volumes? that is both a general Linux best practice as well as the backup method for KVM.
Then you linked a python script that does exactly what I said. Suspends the VM and takes a snapshot of the logical volume.
My point is, why are you saying here that it makes no sense?
Does it? Who really feels that a "you have to shut down the host" backup method is generally valid? I think most companies would consider that a bit of a fail. Does it work? Sure. Does it make sense? Not really. Once you are shutting down the VM, just copy the files, no need to snap the whole underlying volume. Are there benefits to the snap? A little, but you are getting into a less than ideal zone here. I'm certainly not pushing that.
So did it make sense before, no. Does it change with BtrFS, no. Are there edge cases where you can make it work or it might make sense in niche cases? Yes, but not in general ones.
To someone else I was talking about the importance of having LVM and taking a snap inside Linux is quite different than taking one under it. And it is the backup method of KVM, which many people want. That it is there and that I recommend it are not related.
I feel that everything that you quoted from me is consistent. Different use cases. You have to read into it something that I didn't say to have a conflict. KVM has LVM as its snap / image backup method. I rarely would recommend using that. I would never consider running a production system without snap capacity, though. It is a best practice to keep LVM in Linux. Using snaps without quiescence as a backup system is not ideal regardless of if it is LVM or BtrFS, yes you can force quintessence by offlining the system, but I've not recommended that either, not outside of an emergency measure.
-
I think the gap here might be the difference between recommending having a capacity to use a mechanism versus recommending using that mechanism in a given manner.
-
@scottalanmiller said in Linux: BtrFS:
@stacksofplates said in Linux: BtrFS:
Did it make sense before with LVM and XFS? No, and the reason is because you can't snap outside of the VM because there isn't enough information to know if the data is quiesced.
It does make sense. You can easily suspend the VM and take a LV snapshot.
You've also said this to someone else which is what I was referring to:
Why are you not using logical volumes? that is both a general Linux best practice as well as the backup method for KVM.
Then you linked a python script that does exactly what I said. Suspends the VM and takes a snapshot of the logical volume.
My point is, why are you saying here that it makes no sense?
Does it? Who really feels that a "you have to shut down the host" backup method is generally valid? I think most companies would consider that a bit of a fail. Does it work? Sure. Does it make sense? Not really. Once you are shutting down the VM, just copy the files, no need to snap the whole underlying volume. Are there benefits to the snap? A little, but you are getting into a less than ideal zone here. I'm certainly not pushing that.
So did it make sense before, no. Does it change with BtrFS, no. Are there edge cases where you can make it work or it might make sense in niche cases? Yes, but not in general ones.
To someone else I was talking about the importance of having LVM and taking a snap inside Linux is quite different than taking one under it. And it is the backup method of KVM, which many people want. That it is there and that I recommend it are not related.
I feel that everything that you quoted from me is consistent. Different use cases. You have to read into it something that I didn't say to have a conflict. KVM has LVM as its snap / image backup method. I rarely would recommend using that. I would never consider running a production system without snap capacity, though. It is a best practice to keep LVM in Linux. Using snaps without quiescence as a backup system is not ideal regardless of if it is LVM or BtrFS, yes you can force quintessence by offlining the system, but I've not recommended that either, not outside of an emergency measure.
You're not "shutting down the host". It's suspended, completely different. It's perfectly valid because when you take an internal snapshot with a qcow2 file, it does exactly the same thing.
You were not talking about using internal snapshots at all. He was asking how to spin up a backup of a VM and you recommended using logical volume snapshots. Don't try to change what you said. It wasn't an inconsistent quote, you said exactly what I posted in the same manner we are talking about here.
-
@stacksofplates said in Linux: BtrFS:
You were not talking about using internal snapshots at all. He was asking how to spin up a backup of a VM and you recommended using logical volume snapshots. Don't try to change what you said. It wasn't an inconsistent quote, you said exactly what I posted in the same manner we are talking about here.
You only posted my portion of it, what's the whole thing?
