Transparent compression on folders in EXT4/XFS etc?
-
Already when the dinosaurs roamed the world Windows NT had compression on folders in NTFS that were transparent to the user/programs. Even MS-DOS had compression but the entire partition I believe.
Is there a way to do this in linux with standard journaling file systems like EXT4, XFS etc?
-
Nope. See this: https://en.wikipedia.org/wiki/Comparison_of_file_systems#Allocation_and_layout_policies
Your only options are Btrfs or ZFS. -
@marcinozga said in Transparent compression on folders in EXT4/XFS etc?:
Nope. See this: https://en.wikipedia.org/wiki/Comparison_of_file_systems#Allocation_and_layout_policies
Your only options are Btrfs or ZFS.ZFS being the obvious choice for everything
-
Sadly, no. Something that just got left out of the mainline journaled FS world.
-
Too bad that wasn't possible.
BTW looking around I found that Facebook announced last year that they are running btrfs in production on millions of servers.
https://code.fb.com/open-source/linux/ -
@Pete-S said in Transparent compression on folders in EXT4/XFS etc?:
BTW looking around I found that Facebook announced last year that they are running btrfs in production on millions of servers
Have been for quite a while now. They used it quite early.
-
@scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:
@Pete-S said in Transparent compression on folders in EXT4/XFS etc?:
BTW looking around I found that Facebook announced last year that they are running btrfs in production on millions of servers
Have been for quite a while now. They used it quite early.
From what I understood they have been using it since 2014 or so, but not on the massive scale they are now.
-
@Pete-S said in Transparent compression on folders in EXT4/XFS etc?:
@scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:
@Pete-S said in Transparent compression on folders in EXT4/XFS etc?:
BTW looking around I found that Facebook announced last year that they are running btrfs in production on millions of servers
Have been for quite a while now. They used it quite early.
From what I understood they have been using it since 2014 or so, but not on the massive scale they are now.
Their overall scale is bigger now, too. But yeah, they've been moving it from fringe systems to core ones for a while now. Not sure what features of it are deemed important to them.
-
@Pete-S said in Transparent compression on folders in EXT4/XFS etc?:
Too bad that wasn't possible.
BTW looking around I found that Facebook announced last year that they are running btrfs in production on millions of servers.
https://code.fb.com/open-source/linux/I wonder how often they have to restore a system from backup because BtrFS decided to crap out...
Edit: Maybe they can fix that.
-
I'm actually toying around with ZFS at the house now using it's lz4 compression.... Getting ~1.8x compression...
Doesn't seem to affect the speed of anything either.
root@bigcomputer ~ # zfs get compress NAME PROPERTY VALUE SOURCE TestVM compression lz4 local TestVM/Storage compression lz4 inherited from TestVM root@bigcomputer ~ # zfs get compressratio NAME PROPERTY VALUE SOURCE TestVM compressratio 1.87x - TestVM/Storage compressratio 1.87x - root@bigcomputer ~ # -
@dafyre said in Transparent compression on folders in EXT4/XFS etc?:
Doesn't seem to affect the speed of anything either.
SSD or Winchester?
-
@scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:
@dafyre said in Transparent compression on folders in EXT4/XFS etc?:
Doesn't seem to affect the speed of anything either.
SSD or Winchester?
Winchester.
Edit: I'm going to put my Zabbix instance on it later and see how it does.
-
@dafyre said in Transparent compression on folders in EXT4/XFS etc?:
@scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:
@dafyre said in Transparent compression on folders in EXT4/XFS etc?:
Doesn't seem to affect the speed of anything either.
SSD or Winchester?
Winchester.
Oh, would expect it to speed up.
-
@dafyre said in Transparent compression on folders in EXT4/XFS etc?:
Edit: I'm going to put my Zabbix instance on it later and see how it does.
Databases should not be compressed!
-
@scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:
@dafyre said in Transparent compression on folders in EXT4/XFS etc?:
Edit: I'm going to put my Zabbix instance on it later and see how it does.
Databases should not be compressed!
In an ideal world.
The Database itself is not compressed... the entire VM is.
9GB down to 3.2GB. The system runs pretty good. I can't say it's noticeably faster, but it's definitely not slower.
-
@dafyre said in Transparent compression on folders in EXT4/XFS etc?:
The Database itself is not compressed... the entire VM is.
Same issues
It doesn't care where the compression sits. -
@dafyre said in Transparent compression on folders in EXT4/XFS etc?:
9GB down to 3.2GB. The system runs pretty good. I can't say it's noticeably faster, but it's definitely not slower.
That's cool, a lot of savings for sure.
-
@scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:
@dafyre said in Transparent compression on folders in EXT4/XFS etc?:
Edit: I'm going to put my Zabbix instance on it later and see how it does.
Databases should not be compressed!
Details as to why databases should not be compressed?
-
@scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:
@dafyre said in Transparent compression on folders in EXT4/XFS etc?:
@scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:
@dafyre said in Transparent compression on folders in EXT4/XFS etc?:
Doesn't seem to affect the speed of anything either.
SSD or Winchester?
Winchester.
Oh, would expect it to speed up.
So far the only side effects that I'm noticing are the increased amounts of free space and a a (significant?) increase in the number or processes on the system... not necessarily system load, either... I wouldn't have noticed it if a Zabbix trigger hadn't hollered about it.
-
@dafyre said in Transparent compression on folders in EXT4/XFS etc?:
@scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:
@dafyre said in Transparent compression on folders in EXT4/XFS etc?:
Edit: I'm going to put my Zabbix instance on it later and see how it does.
Databases should not be compressed!
Details as to why databases should not be compressed?
Basically because they are always open and written to incrementally. They aren't loaded and rewriteen like most files are. And they tend to be very large, so a very intensive usage pattern.
