ML
    • Recent
    • Categories
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Transparent compression on folders in EXT4/XFS etc?

    Scheduled Pinned Locked Moved IT Discussion
    compressionext4xfszfsbtrfs
    23 Posts 5 Posters 20.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • 1
      1337
      last edited by 1337

      Too bad that wasn't possible.

      BTW looking around I found that Facebook announced last year that they are running btrfs in production on millions of servers.
      https://code.fb.com/open-source/linux/

      scottalanmillerS dafyreD 2 Replies Last reply Reply Quote 0
      • scottalanmillerS
        scottalanmiller @1337
        last edited by

        @Pete-S said in Transparent compression on folders in EXT4/XFS etc?:

        BTW looking around I found that Facebook announced last year that they are running btrfs in production on millions of servers

        Have been for quite a while now. They used it quite early.

        1 1 Reply Last reply Reply Quote 0
        • 1
          1337 @scottalanmiller
          last edited by 1337

          @scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:

          @Pete-S said in Transparent compression on folders in EXT4/XFS etc?:

          BTW looking around I found that Facebook announced last year that they are running btrfs in production on millions of servers

          Have been for quite a while now. They used it quite early.

          From what I understood they have been using it since 2014 or so, but not on the massive scale they are now.

          scottalanmillerS 1 Reply Last reply Reply Quote 0
          • scottalanmillerS
            scottalanmiller @1337
            last edited by

            @Pete-S said in Transparent compression on folders in EXT4/XFS etc?:

            @scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:

            @Pete-S said in Transparent compression on folders in EXT4/XFS etc?:

            BTW looking around I found that Facebook announced last year that they are running btrfs in production on millions of servers

            Have been for quite a while now. They used it quite early.

            From what I understood they have been using it since 2014 or so, but not on the massive scale they are now.

            Their overall scale is bigger now, too. But yeah, they've been moving it from fringe systems to core ones for a while now. Not sure what features of it are deemed important to them.

            1 Reply Last reply Reply Quote 0
            • dafyreD
              dafyre @1337
              last edited by dafyre

              @Pete-S said in Transparent compression on folders in EXT4/XFS etc?:

              Too bad that wasn't possible.

              BTW looking around I found that Facebook announced last year that they are running btrfs in production on millions of servers.
              https://code.fb.com/open-source/linux/

              I wonder how often they have to restore a system from backup because BtrFS decided to crap out...

              Edit: Maybe they can fix that.

              1 Reply Last reply Reply Quote 1
              • dafyreD
                dafyre
                last edited by

                I'm actually toying around with ZFS at the house now using it's lz4 compression.... Getting ~1.8x compression...

                Doesn't seem to affect the speed of anything either.

                root@bigcomputer ~ # zfs get compress
                NAME            PROPERTY     VALUE     SOURCE
                TestVM          compression  lz4       local
                TestVM/Storage  compression  lz4       inherited from TestVM
                
                root@bigcomputer ~ # zfs get compressratio
                NAME            PROPERTY       VALUE  SOURCE
                TestVM          compressratio  1.87x  -
                TestVM/Storage  compressratio  1.87x  -
                root@bigcomputer ~ #
                
                scottalanmillerS 1 Reply Last reply Reply Quote 0
                • scottalanmillerS
                  scottalanmiller @dafyre
                  last edited by

                  @dafyre said in Transparent compression on folders in EXT4/XFS etc?:

                  Doesn't seem to affect the speed of anything either.

                  SSD or Winchester?

                  dafyreD 1 Reply Last reply Reply Quote 0
                  • dafyreD
                    dafyre @scottalanmiller
                    last edited by dafyre

                    @scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:

                    @dafyre said in Transparent compression on folders in EXT4/XFS etc?:

                    Doesn't seem to affect the speed of anything either.

                    SSD or Winchester?

                    Winchester.

                    Edit: I'm going to put my Zabbix instance on it later and see how it does.

                    scottalanmillerS 2 Replies Last reply Reply Quote 0
                    • scottalanmillerS
                      scottalanmiller @dafyre
                      last edited by

                      @dafyre said in Transparent compression on folders in EXT4/XFS etc?:

                      @scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:

                      @dafyre said in Transparent compression on folders in EXT4/XFS etc?:

                      Doesn't seem to affect the speed of anything either.

                      SSD or Winchester?

                      Winchester.

                      Oh, would expect it to speed up.

                      dafyreD 1 Reply Last reply Reply Quote 0
                      • scottalanmillerS
                        scottalanmiller @dafyre
                        last edited by

                        @dafyre said in Transparent compression on folders in EXT4/XFS etc?:

                        Edit: I'm going to put my Zabbix instance on it later and see how it does.

                        Databases should not be compressed!

                        dafyreD 2 Replies Last reply Reply Quote 0
                        • dafyreD
                          dafyre @scottalanmiller
                          last edited by

                          @scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:

                          @dafyre said in Transparent compression on folders in EXT4/XFS etc?:

                          Edit: I'm going to put my Zabbix instance on it later and see how it does.

                          Databases should not be compressed!

                          In an ideal world. 🙂

                          The Database itself is not compressed... the entire VM is. 🙂

                          9GB down to 3.2GB. The system runs pretty good. I can't say it's noticeably faster, but it's definitely not slower.

                          scottalanmillerS 2 Replies Last reply Reply Quote 0
                          • scottalanmillerS
                            scottalanmiller @dafyre
                            last edited by

                            @dafyre said in Transparent compression on folders in EXT4/XFS etc?:

                            The Database itself is not compressed... the entire VM is.

                            Same issues 🙂 It doesn't care where the compression sits.

                            1 Reply Last reply Reply Quote 0
                            • scottalanmillerS
                              scottalanmiller @dafyre
                              last edited by

                              @dafyre said in Transparent compression on folders in EXT4/XFS etc?:

                              9GB down to 3.2GB. The system runs pretty good. I can't say it's noticeably faster, but it's definitely not slower.

                              That's cool, a lot of savings for sure.

                              1 Reply Last reply Reply Quote 0
                              • dafyreD
                                dafyre @scottalanmiller
                                last edited by

                                @scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:

                                @dafyre said in Transparent compression on folders in EXT4/XFS etc?:

                                Edit: I'm going to put my Zabbix instance on it later and see how it does.

                                Databases should not be compressed!

                                Details as to why databases should not be compressed?

                                scottalanmillerS 1 Reply Last reply Reply Quote 0
                                • dafyreD
                                  dafyre @scottalanmiller
                                  last edited by

                                  @scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:

                                  @dafyre said in Transparent compression on folders in EXT4/XFS etc?:

                                  @scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:

                                  @dafyre said in Transparent compression on folders in EXT4/XFS etc?:

                                  Doesn't seem to affect the speed of anything either.

                                  SSD or Winchester?

                                  Winchester.

                                  Oh, would expect it to speed up.

                                  So far the only side effects that I'm noticing are the increased amounts of free space and a a (significant?) increase in the number or processes on the system... not necessarily system load, either... I wouldn't have noticed it if a Zabbix trigger hadn't hollered about it.

                                  1 Reply Last reply Reply Quote 0
                                  • scottalanmillerS
                                    scottalanmiller @dafyre
                                    last edited by

                                    @dafyre said in Transparent compression on folders in EXT4/XFS etc?:

                                    @scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:

                                    @dafyre said in Transparent compression on folders in EXT4/XFS etc?:

                                    Edit: I'm going to put my Zabbix instance on it later and see how it does.

                                    Databases should not be compressed!

                                    Details as to why databases should not be compressed?

                                    Basically because they are always open and written to incrementally. They aren't loaded and rewriteen like most files are. And they tend to be very large, so a very intensive usage pattern.

                                    dafyreD 1 Reply Last reply Reply Quote 0
                                    • dafyreD
                                      dafyre @scottalanmiller
                                      last edited by

                                      @scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:

                                      @dafyre said in Transparent compression on folders in EXT4/XFS etc?:

                                      @scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:

                                      @dafyre said in Transparent compression on folders in EXT4/XFS etc?:

                                      Edit: I'm going to put my Zabbix instance on it later and see how it does.

                                      Databases should not be compressed!

                                      Details as to why databases should not be compressed?

                                      Basically because they are always open and written to incrementally. They aren't loaded and rewriteen like most files are. And they tend to be very large, so a very intensive usage pattern.

                                      True. But this compression is being done on the Host OS, not inside the Zabbix VM. I wonder what kind of strangeness this can cause. I don't have a lot of traffic on this particular server.

                                      scottalanmillerS 1 Reply Last reply Reply Quote 0
                                      • scottalanmillerS
                                        scottalanmiller @dafyre
                                        last edited by

                                        @dafyre said in Transparent compression on folders in EXT4/XFS etc?:

                                        @scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:

                                        @dafyre said in Transparent compression on folders in EXT4/XFS etc?:

                                        @scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:

                                        @dafyre said in Transparent compression on folders in EXT4/XFS etc?:

                                        Edit: I'm going to put my Zabbix instance on it later and see how it does.

                                        Databases should not be compressed!

                                        Details as to why databases should not be compressed?

                                        Basically because they are always open and written to incrementally. They aren't loaded and rewriteen like most files are. And they tend to be very large, so a very intensive usage pattern.

                                        True. But this compression is being done on the Host OS, not inside the Zabbix VM. I wonder what kind of strangeness this can cause. I don't have a lot of traffic on this particular server.

                                        That doesn't affect anything. Compression is compression.

                                        dafyreD 1 Reply Last reply Reply Quote 0
                                        • dafyreD
                                          dafyre @scottalanmiller
                                          last edited by

                                          @scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:

                                          @dafyre said in Transparent compression on folders in EXT4/XFS etc?:

                                          @scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:

                                          @dafyre said in Transparent compression on folders in EXT4/XFS etc?:

                                          @scottalanmiller said in Transparent compression on folders in EXT4/XFS etc?:

                                          @dafyre said in Transparent compression on folders in EXT4/XFS etc?:

                                          Edit: I'm going to put my Zabbix instance on it later and see how it does.

                                          Databases should not be compressed!

                                          Details as to why databases should not be compressed?

                                          Basically because they are always open and written to incrementally. They aren't loaded and rewriteen like most files are. And they tend to be very large, so a very intensive usage pattern.

                                          True. But this compression is being done on the Host OS, not inside the Zabbix VM. I wonder what kind of strangeness this can cause. I don't have a lot of traffic on this particular server.

                                          That doesn't affect anything. Compression is compression.

                                          I'll find out what kind of performance hits I take with it on ZFS. So far, I'm seeing some nice space savings and no problems with anything else.

                                          1 Reply Last reply Reply Quote 0
                                          • 1
                                          • 2
                                          • 1 / 2
                                          • First post
                                            Last post