KVM Virtual Machine Boot Order



  • Hi All,

    So I have a question for the KVM folks out here... How do you manage the order that your VMs start in?

    For instance, at home, my KVM server boots and several other VMs start before my DHCP server, which causes them to not get an IP address.

    I have an idea on how to manage the boot order...I just want to know what others are doing first.



  • never got the issue on centos at work, but this trick of "dummy service" maybe can solve your problem too.

    just create a dummy services which waits for the network service to be up and let it be started Before libvirtd.
    In this way you do not modify original service unit files but you can add extra dependency between network and libvirt.



  • @matteo-nunziati said in KVM Virtual Machine Boot Order:

    never got the issue on centos at work, but this trick of "dummy service" maybe can solve your problem too.

    just create a dummy services which waits for the network service to be up and let it be started Before libvirtd.
    In this way you do not modify original service unit files but you can add extra dependency between network and libvirt.

    That's an interesting take on it. I'll check that out... I've actually been tinkering with doing it a different way. (https://gitlab.com/dafyre/kvm-boot-order)...



  • Is this really not a default option in KVM? I've played with KVM only briefly so haven't had the chance to notice.



  • @dustinb3403 said in KVM Virtual Machine Boot Order:

    Is this really not a default option in KVM? I've played with KVM only briefly so haven't had the chance to notice.

    Not that I'm aware of... Come to think of it, I'm not even sure this is an option in VMware, is it?



  • @dafyre said in KVM Virtual Machine Boot Order:

    @dustinb3403 said in KVM Virtual Machine Boot Order:

    Is this really not a default option in KVM? I've played with KVM only briefly so haven't had the chance to notice.

    Not that I'm aware of... Come to think of it, I'm not even sure this is an option in VMware, is it?

    It is.



  • @dafyre said in KVM Virtual Machine Boot Order:

    @dustinb3403 said in KVM Virtual Machine Boot Order:

    Is this really not a default option in KVM? I've played with KVM only briefly so haven't had the chance to notice.

    Not that I'm aware of... Come to think of it, I'm not even sure this is an option in VMware, is it?

    There is a boot delay, not seeing a boot order specifically.



  • @dustinb3403 said in KVM Virtual Machine Boot Order:

    @dafyre said in KVM Virtual Machine Boot Order:

    @dustinb3403 said in KVM Virtual Machine Boot Order:

    Is this really not a default option in KVM? I've played with KVM only briefly so haven't had the chance to notice.

    Not that I'm aware of... Come to think of it, I'm not even sure this is an option in VMware, is it?

    There is a boot delay, not seeing a boot order specifically.

    In Hyper-V. It is a boot delay. All guests either boot or not on startup. A binary choice. Then you additionally can choose to delay them.

    I thought VMWare had a boot order and boot delays. when I last used it (VMWare 5.0)



  • @jaredbusch said in KVM Virtual Machine Boot Order:

    @dustinb3403 said in KVM Virtual Machine Boot Order:

    @dafyre said in KVM Virtual Machine Boot Order:

    @dustinb3403 said in KVM Virtual Machine Boot Order:

    Is this really not a default option in KVM? I've played with KVM only briefly so haven't had the chance to notice.

    Not that I'm aware of... Come to think of it, I'm not even sure this is an option in VMware, is it?

    There is a boot delay, not seeing a boot order specifically.

    In Hyper-V. It is a boot delay. All guests either boot or not on startup. A binary choice. Then you additionally can choose to delay them.

    I thought VMWare had a boot order and boot delays. when I last used it (VMWare 5.0)

    You can define a boot order/priority so that one thing starts before others. There is then a boot delay that you can throw at it as well.



  • On 6.5 I can find the option for boot delay, but I'm not seeing a boot order. I'm probably just looking in the wrong place but I thought I've seen it before as well.



  • @dustinb3403 said in KVM Virtual Machine Boot Order:

    On 6.5 I can find the option for boot delay, but I'm not seeing a boot order. I'm probably just looking in the wrong place but I thought I've seen it before as well.

    If you mean VMware 6.5, Select Host -> Configure -> Virtual Machines. Put your selected VM's in Automatic in the order you want them to start up and the delay between each startup.



  • Uncomment START_DELAY=0 in /etc/sysconfig/libvirt-guests file.

    https://access.redhat.com/discussions/685243
    0_1533138645107_8657357f-89b9-4565-a0ba-c6a64389a585-image.png

    0_1533138783472_51e96b88-60ae-443b-8cbc-2c3ebac10574-image.png



  • You can also use crontab and virsh and make them not just have an order, but make them wait for each other as well.



  • @pmoncho said in KVM Virtual Machine Boot Order:

    @dustinb3403 said in KVM Virtual Machine Boot Order:

    On 6.5 I can find the option for boot delay, but I'm not seeing a boot order. I'm probably just looking in the wrong place but I thought I've seen it before as well.

    If you mean VMware 6.5, Select Host -> Configure -> Virtual Machines. Put your selected VM's in Automatic in the order you want them to start up and the delay between each startup.

    I don't see that anywhere in my installation, probably because we're using the free version of ESXi. . . (yes I know)



  • I've only ever been aware of the autostart option in Virtual Machine Manager:

    0_1533144456858_4b552e40-b8eb-49e7-957a-d54ec5596ee8-image.png

    Or via virsh autostart:

    0_1533144495370_c7baea6a-f3f0-4e0f-bfbb-e3b08cab477b-image.png

    So perhaps the easiest way to do this, since there's no built-in function like Hyper-V's, is to disable autostart of all VMs, and then create a bash startup script that starts the VMs one by one, using virsh start, and then adding a delay in between.



  • @obsolesce That sounds like the exact thing I want to do all day. . . create a script to start my VM's in sequence. . . oh and update said script every time I add or remove a VM.

    Rather than having a boot sequence/priority function built in. . .



  • @dustinb3403 said in KVM Virtual Machine Boot Order:

    @obsolesce That sounds like the exact thing I want to do all day. . . create a script to start my VM's in sequence. . .

    Rather than having a boot sequence/priority function built in. . .

    I know exactly what you mean... I had the timing set up good in Hyper-V that has about 70 VMs... but the automatic startup was not reliable at all. So I created a PowerShell script to do it on boot, which worked great. But was just one more thing to worry about when a new VM was created. Non-existant VMs didn't matter, it would throw an error in PowerShell, but would continue on which wasn't an issue.



  • Don't get me wrong, I don't mind making a script to do things, but it's just an added pain, and if you set it up and forget it then the code becomes cobwebbed in your brain, you forget, have an issue, have to remember what you did. It's an ordeal that really shouldn't exist.

    Boot priority seems like one of those things that should've been included from the get-go. Like seatbelts in cars are today. . .



  • @dustinb3403 said in KVM Virtual Machine Boot Order:

    Don't get me wrong, I don't mind making a script to do things, but it's just an added pain, and if you set it up and forget it then the code becomes cobwebbed in your brain, you forget, have an issue, have to remember what you did. It's an ordeal that really shouldn't exist.

    Boot priority seems like one of those things that should've been included from the get-go. Like seatbelts in cars are today. . .

    Yeah I agree. I don't use the script anymore. I do it all manually now. It's just less to worry about when you can select multiple, right-click, turn on...

    I have the very few important ones set to automatically turn on first, like DHCP, AD, and some of the databases. Hyper-V can handle doing a few without issue.



  • Just as a note, this problem of startup order appeared to have been an issue since the inception of KVM and even before. It's as if the devs simply thing a startup delay or manually starting your VM's is sufficient. . .



  • @dustinb3403 said in KVM Virtual Machine Boot Order:

    Don't get me wrong, I don't mind making a script to do things, but it's just an added pain, and if you set it up and forget it then the code becomes cobwebbed in your brain, you forget, have an issue, have to remember what you did. It's an ordeal that really shouldn't exist.

    Boot priority seems like one of those things that should've been included from the get-go. Like seatbelts in cars are today. . .

    That's one of the reasons I put a file in /etc/default -- so we don't have to modify the script itself every time a VM is added or removed. Yes, we do have to touch a file, but that shouldn't be too difficult... right?

    I'm not overly interested in error handling or anything like that with this script -- if the VM boots, then it will work. If it doesn't boot, then you have larger issues to deal with, lol.



  • @dustinb3403 said in KVM Virtual Machine Boot Order:

    @pmoncho said in KVM Virtual Machine Boot Order:

    @dustinb3403 said in KVM Virtual Machine Boot Order:

    On 6.5 I can find the option for boot delay, but I'm not seeing a boot order. I'm probably just looking in the wrong place but I thought I've seen it before as well.

    If you mean VMware 6.5, Select Host -> Configure -> Virtual Machines. Put your selected VM's in Automatic in the order you want them to start up and the delay between each startup.

    I don't see that anywhere in my installation, probably because we're using the free version of ESXi. . . (yes I know)

    My bad. I assumed you had vCenter. I should have asked. I don't see any way on the free edition.



  • @dustinb3403 said in KVM Virtual Machine Boot Order:

    Just as a note, this problem of startup order appeared to have been an issue since the inception of KVM and even before. It's as if the devs simply thing a startup delay or manually starting your VM's is sufficient. . .

    It's really not a big deal though...

    If you're a smaller shop, it takes no time to manually start your VMs.

    If you're huge, you just set them all to start at boot because you have entire hosts dedicated to one type of VM and it doesn't matter in what order.

    If medium sized, it's still not a big deal. I don't mind anyways. If I reboot the main host, I'm there watching it anyways. So it's not an issue to start up the DHCP server first, and a few other initial VMs, then the rest in one shot.



  • @obsolesce but it could be an issue if your DHCP server starts after other VM's. Or a database server that is dependant on other VM's being up and accessible first.

    There is truly a case for needing a startup priority here.

    I wonder what Amazon does. . .



  • @dustinb3403 said in KVM Virtual Machine Boot Order:

    @obsolesce but it could be an issue if your DHCP server starts after other VM's. Or a database server that is dependant on other VM's being up and accessible first.

    There is truly a case for needing a startup priority here.

    I wonder what Amazon does. . .

    My point was that, Amazon doesn't have DB, DHCP servers, and other app VMs that depend on them on the same hosts.

    And those companies who do are small enough to just do it manually with minimal effort.

    I have a host with 70 VMs on it, it takes seconds to start the AD and DHCP server first, then a few database servers.



  • @obsolesce said in KVM Virtual Machine Boot Order:

    @dustinb3403 said in KVM Virtual Machine Boot Order:

    @obsolesce but it could be an issue if your DHCP server starts after other VM's. Or a database server that is dependant on other VM's being up and accessible first.

    There is truly a case for needing a startup priority here.

    I wonder what Amazon does. . .

    My point was that, Amazon doesn't have hosts with DB servers and DHCP servers and other apps that depend on them on the same hosts.

    And those companies who do are small enough to just do it manually with minimal effort.

    I have a host with 70 VMs on it, it takes seconds to start the AD and DHCP server first, then a few database servers.

    How are you starting the remaining 60 VM's?



  • @dustinb3403 said in KVM Virtual Machine Boot Order:

    @obsolesce said in KVM Virtual Machine Boot Order:

    @dustinb3403 said in KVM Virtual Machine Boot Order:

    @obsolesce but it could be an issue if your DHCP server starts after other VM's. Or a database server that is dependant on other VM's being up and accessible first.

    There is truly a case for needing a startup priority here.

    I wonder what Amazon does. . .

    My point was that, Amazon doesn't have hosts with DB servers and DHCP servers and other apps that depend on them on the same hosts.

    And those companies who do are small enough to just do it manually with minimal effort.

    I have a host with 70 VMs on it, it takes seconds to start the AD and DHCP server first, then a few database servers.

    How are you starting the remaining 60 VM's?

    Select-all or shift+click, right-click on them, click start.

    Edit, this is on Hyper-V.



  • @obsolesce that's annoying. . .

    Edit: still annoying but doesn't Hyper-V have startup priority?



  • @dustinb3403 said in KVM Virtual Machine Boot Order:

    @obsolesce that's annoying. . .

    Not really, it's once a month at worst.



  • @dustinb3403 said in KVM Virtual Machine Boot Order:

    @obsolesce that's annoying. . .

    Edit: still annoying but doesn't Hyper-V have startup priority?

    No it hasnt until I checked 6 months ago. Only way is build a delay schema by hand and assign delays to VM batch start them in layers