ML
    • Register
    • Login
    • Search
    • Recent
    • Categories
    • Tags
    • Popular
    • Users
    • Groups

    Patching systems - how should you do this?

    IT Discussion
    4
    22
    1605
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Dashrender
      Dashrender last edited by

      So in the DC died thread I suggested the option of storage motioning a VM to another host to allow systems to be patched with no downtime.

      Scott shot that down pretty quickly as something that should not be done during production times.

      Does this then imply that you either

      1. have to have shared storage to live transfer VMs between hosts for patches, or
      2. expect downtime on VMs while a host is updated?

      Are there other options?

      scottalanmiller 1 Reply Last reply Reply Quote 0
      • scottalanmiller
        scottalanmiller @Dashrender last edited by

        @Dashrender said in Patching systems - how should you do this?:

        Does this then imply that you either

        1. have to have shared storage to live transfer VMs between hosts for patches, or
        2. expect downtime on VMs while a host is updated?

        or 3) have an HA application that doesn't have a dependency at that level, like an AD DC.

        Dashrender 1 Reply Last reply Reply Quote 2
        • scottalanmiller
          scottalanmiller last edited by

          But those are pretty much the three options. CAN you storage motion? Sure. Advised? No. Other than that, you need a way to keep services running.

          Ask yourself this, if you need to do patching without downtime (planned) how do you have that need but don't need to protect against unplanned downtime (outages?) If you are protecting against outages, you naturally have the ability to patch, right?

          Dashrender 1 Reply Last reply Reply Quote 0
          • Dashrender
            Dashrender @scottalanmiller last edited by

            @scottalanmiller said in Patching systems - how should you do this?:

            @Dashrender said in Patching systems - how should you do this?:

            Does this then imply that you either

            1. have to have shared storage to live transfer VMs between hosts for patches, or
            2. expect downtime on VMs while a host is updated?

            or 3) have an HA application that doesn't have a dependency at that level, like an AD DC.

            Awesome - exactly what I was looking for.

            scottalanmiller 1 Reply Last reply Reply Quote 1
            • Dashrender
              Dashrender @scottalanmiller last edited by

              @scottalanmiller said in Patching systems - how should you do this?:

              But those are pretty much the three options. CAN you storage motion? Sure. Advised? No. Other than that, you need a way to keep services running.

              Ask yourself this, if you need to do patching without downtime (planned) how do you have that need but don't need to protect against unplanned downtime (outages?) If you are protecting against outages, you naturally have the ability to patch, right?

              Well, this is self serving - but the desire to not have to work outside the normal business hours to accomplish patching was my primary thinking. But that cost, to the company, is probably not worth it.

              scottalanmiller 1 Reply Last reply Reply Quote 0
              • scottalanmiller
                scottalanmiller @Dashrender last edited by

                @Dashrender said in Patching systems - how should you do this?:

                @scottalanmiller said in Patching systems - how should you do this?:

                But those are pretty much the three options. CAN you storage motion? Sure. Advised? No. Other than that, you need a way to keep services running.

                Ask yourself this, if you need to do patching without downtime (planned) how do you have that need but don't need to protect against unplanned downtime (outages?) If you are protecting against outages, you naturally have the ability to patch, right?

                Well, this is self serving - but the desire to not have to work outside the normal business hours to accomplish patching was my primary thinking. But that cost, to the company, is probably not worth it.

                If you have the resources to storage motion, you likely have them to do shared storage too, though. So likely no cost.

                1 Reply Last reply Reply Quote 0
                • scottalanmiller
                  scottalanmiller @Dashrender last edited by

                  @Dashrender said in Patching systems - how should you do this?:

                  @scottalanmiller said in Patching systems - how should you do this?:

                  @Dashrender said in Patching systems - how should you do this?:

                  Does this then imply that you either

                  1. have to have shared storage to live transfer VMs between hosts for patches, or
                  2. expect downtime on VMs while a host is updated?

                  or 3) have an HA application that doesn't have a dependency at that level, like an AD DC.

                  Awesome - exactly what I was looking for.

                  With web servers, for example, this would be behind the load balacing layer. Just remove a server from the LB, patch and add it back in.

                  stacksofplates 1 Reply Last reply Reply Quote 0
                  • stacksofplates
                    stacksofplates @scottalanmiller last edited by

                    @scottalanmiller said in Patching systems - how should you do this?:

                    @Dashrender said in Patching systems - how should you do this?:

                    @scottalanmiller said in Patching systems - how should you do this?:

                    @Dashrender said in Patching systems - how should you do this?:

                    Does this then imply that you either

                    1. have to have shared storage to live transfer VMs between hosts for patches, or
                    2. expect downtime on VMs while a host is updated?

                    or 3) have an HA application that doesn't have a dependency at that level, like an AD DC.

                    Awesome - exactly what I was looking for.

                    With web servers, for example, this would be behind the load balacing layer. Just remove a server from the LB, patch and add it back in.

                    Or don't patch and just spin up a new one with the data store somewhere else and kill the old one.

                    scottalanmiller 1 Reply Last reply Reply Quote 3
                    • scottalanmiller
                      scottalanmiller @stacksofplates last edited by

                      @stacksofplates said in Patching systems - how should you do this?:

                      @scottalanmiller said in Patching systems - how should you do this?:

                      @Dashrender said in Patching systems - how should you do this?:

                      @scottalanmiller said in Patching systems - how should you do this?:

                      @Dashrender said in Patching systems - how should you do this?:

                      Does this then imply that you either

                      1. have to have shared storage to live transfer VMs between hosts for patches, or
                      2. expect downtime on VMs while a host is updated?

                      or 3) have an HA application that doesn't have a dependency at that level, like an AD DC.

                      Awesome - exactly what I was looking for.

                      With web servers, for example, this would be behind the load balacing layer. Just remove a server from the LB, patch and add it back in.

                      Or don't patch and just spin up a new one with the data store somewhere else and kill the old one.

                      Yes, this is the DevOps model for this.

                      stacksofplates 1 Reply Last reply Reply Quote 1
                      • stacksofplates
                        stacksofplates @scottalanmiller last edited by

                        @scottalanmiller said in Patching systems - how should you do this?:

                        @stacksofplates said in Patching systems - how should you do this?:

                        @scottalanmiller said in Patching systems - how should you do this?:

                        @Dashrender said in Patching systems - how should you do this?:

                        @scottalanmiller said in Patching systems - how should you do this?:

                        @Dashrender said in Patching systems - how should you do this?:

                        Does this then imply that you either

                        1. have to have shared storage to live transfer VMs between hosts for patches, or
                        2. expect downtime on VMs while a host is updated?

                        or 3) have an HA application that doesn't have a dependency at that level, like an AD DC.

                        Awesome - exactly what I was looking for.

                        With web servers, for example, this would be behind the load balacing layer. Just remove a server from the LB, patch and add it back in.

                        Or don't patch and just spin up a new one with the data store somewhere else and kill the old one.

                        Yes, this is the DevOps model for this.

                        So say you have 5 web servers running. Would you mount the data store from an NFS export or would you run something like Gluster, GFS2, etc across each physical server that the web servers are on?

                        dafyre scottalanmiller 2 Replies Last reply Reply Quote 0
                        • dafyre
                          dafyre @stacksofplates last edited by

                          @stacksofplates said in Patching systems - how should you do this?:

                          @scottalanmiller said in Patching systems - how should you do this?:

                          @stacksofplates said in Patching systems - how should you do this?:

                          @scottalanmiller said in Patching systems - how should you do this?:

                          @Dashrender said in Patching systems - how should you do this?:

                          @scottalanmiller said in Patching systems - how should you do this?:

                          @Dashrender said in Patching systems - how should you do this?:

                          Does this then imply that you either

                          1. have to have shared storage to live transfer VMs between hosts for patches, or
                          2. expect downtime on VMs while a host is updated?

                          or 3) have an HA application that doesn't have a dependency at that level, like an AD DC.

                          Awesome - exactly what I was looking for.

                          With web servers, for example, this would be behind the load balacing layer. Just remove a server from the LB, patch and add it back in.

                          Or don't patch and just spin up a new one with the data store somewhere else and kill the old one.

                          Yes, this is the DevOps model for this.

                          So say you have 5 web servers running. Would you mount the data store from an NFS export or would you run something like Gluster, GFS2, etc across each physical server that the web servers are on?

                          That would depend on your goals. Personally, I'd use a Gluster setup, or either rsync between the servers...

                          Without Gluster or rsync, you're still dead in the water when your NFS server reboots for updates.

                          With rsync, you run into issues if you website(s) support file uploads.

                          stacksofplates 1 Reply Last reply Reply Quote 1
                          • stacksofplates
                            stacksofplates @dafyre last edited by

                            @dafyre said in Patching systems - how should you do this?:

                            @stacksofplates said in Patching systems - how should you do this?:

                            @scottalanmiller said in Patching systems - how should you do this?:

                            @stacksofplates said in Patching systems - how should you do this?:

                            @scottalanmiller said in Patching systems - how should you do this?:

                            @Dashrender said in Patching systems - how should you do this?:

                            @scottalanmiller said in Patching systems - how should you do this?:

                            @Dashrender said in Patching systems - how should you do this?:

                            Does this then imply that you either

                            1. have to have shared storage to live transfer VMs between hosts for patches, or
                            2. expect downtime on VMs while a host is updated?

                            or 3) have an HA application that doesn't have a dependency at that level, like an AD DC.

                            Awesome - exactly what I was looking for.

                            With web servers, for example, this would be behind the load balacing layer. Just remove a server from the LB, patch and add it back in.

                            Or don't patch and just spin up a new one with the data store somewhere else and kill the old one.

                            Yes, this is the DevOps model for this.

                            So say you have 5 web servers running. Would you mount the data store from an NFS export or would you run something like Gluster, GFS2, etc across each physical server that the web servers are on?

                            That would depend on your goals. Personally, I'd use a Gluster setup, or either rsync between the servers...

                            Without Gluster or rsync, you're still dead in the water when your NFS server reboots for updates.

                            With rsync, you run into issues if you website(s) support file uploads.

                            Well I meant an NFS export from some clustered system. Like a Gluster cluster or an Isilon.

                            dafyre 1 Reply Last reply Reply Quote 0
                            • dafyre
                              dafyre @stacksofplates last edited by

                              @stacksofplates said in Patching systems - how should you do this?:

                              @dafyre said in Patching systems - how should you do this?:

                              @stacksofplates said in Patching systems - how should you do this?:

                              @scottalanmiller said in Patching systems - how should you do this?:

                              @stacksofplates said in Patching systems - how should you do this?:

                              @scottalanmiller said in Patching systems - how should you do this?:

                              @Dashrender said in Patching systems - how should you do this?:

                              @scottalanmiller said in Patching systems - how should you do this?:

                              @Dashrender said in Patching systems - how should you do this?:

                              Does this then imply that you either

                              1. have to have shared storage to live transfer VMs between hosts for patches, or
                              2. expect downtime on VMs while a host is updated?

                              or 3) have an HA application that doesn't have a dependency at that level, like an AD DC.

                              Awesome - exactly what I was looking for.

                              With web servers, for example, this would be behind the load balacing layer. Just remove a server from the LB, patch and add it back in.

                              Or don't patch and just spin up a new one with the data store somewhere else and kill the old one.

                              Yes, this is the DevOps model for this.

                              So say you have 5 web servers running. Would you mount the data store from an NFS export or would you run something like Gluster, GFS2, etc across each physical server that the web servers are on?

                              That would depend on your goals. Personally, I'd use a Gluster setup, or either rsync between the servers...

                              Without Gluster or rsync, you're still dead in the water when your NFS server reboots for updates.

                              With rsync, you run into issues if you website(s) support file uploads.

                              Well I meant an NFS export from some clustered system. Like a Gluster cluster or an Isilon.

                              Okay, yeah. In that case if your NFS server is reduntant or fault tolerant whatever you want to call it, then you're in good shape.

                              @scottalanmiller -- how would you build a fault tolerant NFS server for something like this? Two Linux systems + DRBD?

                              stacksofplates scottalanmiller 2 Replies Last reply Reply Quote 0
                              • stacksofplates
                                stacksofplates @dafyre last edited by

                                @dafyre said in Patching systems - how should you do this?:

                                @stacksofplates said in Patching systems - how should you do this?:

                                @dafyre said in Patching systems - how should you do this?:

                                @stacksofplates said in Patching systems - how should you do this?:

                                @scottalanmiller said in Patching systems - how should you do this?:

                                @stacksofplates said in Patching systems - how should you do this?:

                                @scottalanmiller said in Patching systems - how should you do this?:

                                @Dashrender said in Patching systems - how should you do this?:

                                @scottalanmiller said in Patching systems - how should you do this?:

                                @Dashrender said in Patching systems - how should you do this?:

                                Does this then imply that you either

                                1. have to have shared storage to live transfer VMs between hosts for patches, or
                                2. expect downtime on VMs while a host is updated?

                                or 3) have an HA application that doesn't have a dependency at that level, like an AD DC.

                                Awesome - exactly what I was looking for.

                                With web servers, for example, this would be behind the load balacing layer. Just remove a server from the LB, patch and add it back in.

                                Or don't patch and just spin up a new one with the data store somewhere else and kill the old one.

                                Yes, this is the DevOps model for this.

                                So say you have 5 web servers running. Would you mount the data store from an NFS export or would you run something like Gluster, GFS2, etc across each physical server that the web servers are on?

                                That would depend on your goals. Personally, I'd use a Gluster setup, or either rsync between the servers...

                                Without Gluster or rsync, you're still dead in the water when your NFS server reboots for updates.

                                With rsync, you run into issues if you website(s) support file uploads.

                                Well I meant an NFS export from some clustered system. Like a Gluster cluster or an Isilon.

                                Okay, yeah. In that case if your NFS server is reduntant or fault tolerant whatever you want to call it, then you're in good shape.

                                @scottalanmiller -- how would you build a fault tolerant NFS server for something like this? Two Linux systems + DRBD?

                                I would prob use file based rather than block. With Gluster or GFS2 you can have 3 nodes so if you take one physical machine down for an update you don't have to worry about the other going down.

                                dafyre 1 Reply Last reply Reply Quote 2
                                • dafyre
                                  dafyre @stacksofplates last edited by

                                  @stacksofplates said in Patching systems - how should you do this?:

                                  @dafyre said in Patching systems - how should you do this?:

                                  @stacksofplates said in Patching systems - how should you do this?:

                                  @dafyre said in Patching systems - how should you do this?:

                                  @stacksofplates said in Patching systems - how should you do this?:

                                  @scottalanmiller said in Patching systems - how should you do this?:

                                  @stacksofplates said in Patching systems - how should you do this?:

                                  @scottalanmiller said in Patching systems - how should you do this?:

                                  @Dashrender said in Patching systems - how should you do this?:

                                  @scottalanmiller said in Patching systems - how should you do this?:

                                  @Dashrender said in Patching systems - how should you do this?:

                                  Does this then imply that you either

                                  1. have to have shared storage to live transfer VMs between hosts for patches, or
                                  2. expect downtime on VMs while a host is updated?

                                  or 3) have an HA application that doesn't have a dependency at that level, like an AD DC.

                                  Awesome - exactly what I was looking for.

                                  With web servers, for example, this would be behind the load balacing layer. Just remove a server from the LB, patch and add it back in.

                                  Or don't patch and just spin up a new one with the data store somewhere else and kill the old one.

                                  Yes, this is the DevOps model for this.

                                  So say you have 5 web servers running. Would you mount the data store from an NFS export or would you run something like Gluster, GFS2, etc across each physical server that the web servers are on?

                                  That would depend on your goals. Personally, I'd use a Gluster setup, or either rsync between the servers...

                                  Without Gluster or rsync, you're still dead in the water when your NFS server reboots for updates.

                                  With rsync, you run into issues if you website(s) support file uploads.

                                  Well I meant an NFS export from some clustered system. Like a Gluster cluster or an Isilon.

                                  Okay, yeah. In that case if your NFS server is reduntant or fault tolerant whatever you want to call it, then you're in good shape.

                                  @scottalanmiller -- how would you build a fault tolerant NFS server for something like this? Two Linux systems + DRBD?

                                  I would prob use file based rather than block. With Gluster or GFS2 you can have 3 nodes so if you take one physical machine down for an update you don't have to worry about the other going down.

                                  Makes sense.

                                  1 Reply Last reply Reply Quote 0
                                  • scottalanmiller
                                    scottalanmiller @stacksofplates last edited by

                                    @stacksofplates said in Patching systems - how should you do this?:

                                    @scottalanmiller said in Patching systems - how should you do this?:

                                    @stacksofplates said in Patching systems - how should you do this?:

                                    @scottalanmiller said in Patching systems - how should you do this?:

                                    @Dashrender said in Patching systems - how should you do this?:

                                    @scottalanmiller said in Patching systems - how should you do this?:

                                    @Dashrender said in Patching systems - how should you do this?:

                                    Does this then imply that you either

                                    1. have to have shared storage to live transfer VMs between hosts for patches, or
                                    2. expect downtime on VMs while a host is updated?

                                    or 3) have an HA application that doesn't have a dependency at that level, like an AD DC.

                                    Awesome - exactly what I was looking for.

                                    With web servers, for example, this would be behind the load balacing layer. Just remove a server from the LB, patch and add it back in.

                                    Or don't patch and just spin up a new one with the data store somewhere else and kill the old one.

                                    Yes, this is the DevOps model for this.

                                    So say you have 5 web servers running. Would you mount the data store from an NFS export or would you run something like Gluster, GFS2, etc across each physical server that the web servers are on?

                                    Depends, in a lot of cases you would deploy a local image via Ansible or Chef and have it deploy to each node at build time. If you have NFS or something, you introduce a new dependency.

                                    1 Reply Last reply Reply Quote 0
                                    • scottalanmiller
                                      scottalanmiller @dafyre last edited by

                                      @dafyre said in Patching systems - how should you do this?:

                                      @scottalanmiller -- how would you build a fault tolerant NFS server for something like this? Two Linux systems + DRBD?

                                      You could do a two node this way. But for web servers with static files, why not just keep the files local and increase speed, simplify things and reduce complexity?

                                      dafyre 1 Reply Last reply Reply Quote 0
                                      • dafyre
                                        dafyre @scottalanmiller last edited by

                                        @scottalanmiller said in Patching systems - how should you do this?:

                                        @dafyre said in Patching systems - how should you do this?:

                                        @scottalanmiller -- how would you build a fault tolerant NFS server for something like this? Two Linux systems + DRBD?

                                        You could do a two node this way. But for web servers with static files, why not just keep the files local and increase speed, simplify things and reduce complexity?

                                        For systems that are static, sure. But what about something like Wordpress where files actually can be uploaded?

                                        [I realize that may not be the world's greatest example, lol]

                                        scottalanmiller 1 Reply Last reply Reply Quote 0
                                        • scottalanmiller
                                          scottalanmiller @dafyre last edited by

                                          @dafyre said in Patching systems - how should you do this?:

                                          @scottalanmiller said in Patching systems - how should you do this?:

                                          @dafyre said in Patching systems - how should you do this?:

                                          @scottalanmiller -- how would you build a fault tolerant NFS server for something like this? Two Linux systems + DRBD?

                                          You could do a two node this way. But for web servers with static files, why not just keep the files local and increase speed, simplify things and reduce complexity?

                                          For systems that are static, sure. But what about something like Wordpress where files actually can be uploaded?

                                          [I realize that may not be the world's greatest example, lol]

                                          You would store those centrally, but not the main files. Often you would have dedicated image storage in a case that you were going to multi-node scale out web, not storing or serving from the application server. So typically tackled in a completely different way. Either through a CDN that you buy or build yourself.

                                          Just look at ML, getting images to CDN is top priority from the very beginning.

                                          dafyre 1 Reply Last reply Reply Quote 0
                                          • dafyre
                                            dafyre @scottalanmiller last edited by

                                            @scottalanmiller said in Patching systems - how should you do this?:

                                            @dafyre said in Patching systems - how should you do this?:

                                            @scottalanmiller said in Patching systems - how should you do this?:

                                            @dafyre said in Patching systems - how should you do this?:

                                            @scottalanmiller -- how would you build a fault tolerant NFS server for something like this? Two Linux systems + DRBD?

                                            You could do a two node this way. But for web servers with static files, why not just keep the files local and increase speed, simplify things and reduce complexity?

                                            For systems that are static, sure. But what about something like Wordpress where files actually can be uploaded?

                                            [I realize that may not be the world's greatest example, lol]

                                            You would store those centrally, but not the main files. Often you would have dedicated image storage in a case that you were going to multi-node scale out web, not storing or serving from the application server. So typically tackled in a completely different way. Either through a CDN that you buy or build yourself.

                                            Just look at ML, getting images to CDN is top priority from the very beginning.

                                            I wasn't thinking about images, but, I get that idea. I was thinking more along the lines of user submitted uploads... but those could be sent into a database somewhere.

                                            scottalanmiller 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post