ML
    • Recent
    • Categories
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    CentOS 7 VM on Hyper-V losing DHCP assigned address

    IT Discussion
    centos 7 dhcp failed hyper-v chrony
    8
    60
    12.3k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • JaredBuschJ
      JaredBusch
      last edited by

      Well this morning this system was offline again.

      [root@owncloud ~]# ip a sh
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 02:50:56:17:18:7f brd ff:ff:ff:ff:ff:ff

      No IP again. not new.
      It says it is there and has link.

      [root@owncloud ~]# nmcli d
DEVICE  TYPE      STATE      CONNECTION
eth0    ethernet  connected  Wired connection 1
lo      loopback  unmanaged  --

      Let's look at the config file maybe there is something stupid..

      Wait what? WTF? Where is eth0?

      [root@owncloud ~]# ls /etc/sysconfig/network-scripts/ifcfg*
/etc/sysconfig/network-scripts/ifcfg-ens32 /etc/sysconfig/network-scripts/ifcfg-lo

      Could not find it to save my life, but nmtui listed both Wired connection 1 and ens32. Fine. Deleted ens32 as it was the wrong MAC and edited Wired connection 1 since it was the correct MAC. Renamed it to eth0 and exited from nmtui.

      Now I have ifcfg-eth0

      [root@owncloud ~]# ls /etc/sysconfig/network-scripts/ifcfg*
/etc/sysconfig/network-scripts/ifcfg-eth0  /etc/sysconfig/network-scripts/ifcfg-lo

      Let's see if this gets more stable now.

      I have no idea WTF happened here. Nothing on this system has changed for 2 years except yum updates.

      1 Reply Last reply Reply Quote 1
      • JaredBuschJ
        JaredBusch
        last edited by

        Lost the IP again this morning.

        But now that I have the eth0 configured i was able to systemctl restart network and it came up.

        nothing in /var/log/messages for DHCP. Date it correct still.

        So where do I look to resolve this?

        JaredBuschJ 1 Reply Last reply Reply Quote 0
        • JaredBuschJ
          JaredBusch @JaredBusch
          last edited by

          status looked like this again..

          [root@owncloud ~]# ip a sh
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 02:50:56:17:18:7f brd ff:ff:ff:ff:ff:ff
          1 Reply Last reply Reply Quote 0
          • JaredBuschJ
            JaredBusch
            last edited by

            obviously something was making the network work prior to my change yesterday because a reboot would fix it.

            So WTF else makes networking work on CentOS 7 when there is no eth0 config file?

            At this point, I have to assume that process is causing the problem.

            1 Reply Last reply Reply Quote 0
            • dafyreD
              dafyre
              last edited by

              Do you have DHCP server set to give this server a specific IP?

              JaredBuschJ 1 Reply Last reply Reply Quote 0
              • JaredBuschJ
                JaredBusch @dafyre
                last edited by JaredBusch

                @dafyre said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

                Do you have DHCP server set to give this server a specific IP?

                It is a DHCP reservation yes.

                I could obviously make this static, but i hate static, because what if I change the network scope or DNS or something. Too much to manually update in that instance.

                Also this is NOT the first CentOS 7 system I have seen this on. This is the second one used for ownCloud, and I have another that is simply a FTP server for some old backups that has done this in the past.

                All three of the above systems are at different clients sites. with different servers and networks. All three are on Hyper-V though.
                Those two systems have not done this recently, but this system is doing it daily now.

                dafyreD 1 Reply Last reply Reply Quote 0
                • dafyreD
                  dafyre @JaredBusch
                  last edited by

                  @JaredBusch said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

                  @dafyre said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

                  Do you have DHCP server set to give this server a specific IP?

                  It is a DHCP reservation yes.

                  I could obviously make this static, but i hate static, because what if I change the network scope or DNS or something. Too much to manually update in that instance.

                  Also this is NOT the first CentOS 7 system I have seen this on. This is the second one used for ownCloud, and I have another that is simply a FTP server for some old backups that has done this in the past.

                  All three of the above systems are at different clients sites. with different servers and networks. All three are on Hyper-V though.
                  Those two systems have not done this recently, but this system is doing it daily now.

                  Is your DHCP server on the same Hyper-V server?

                  Also, what kind of network switch is it connected to?

                  JaredBuschJ 1 Reply Last reply Reply Quote 0
                  • JaredBuschJ
                    JaredBusch @dafyre
                    last edited by

                    @dafyre said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

                    @JaredBusch said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

                    @dafyre said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

                    Do you have DHCP server set to give this server a specific IP?

                    It is a DHCP reservation yes.

                    I could obviously make this static, but i hate static, because what if I change the network scope or DNS or something. Too much to manually update in that instance.

                    Also this is NOT the first CentOS 7 system I have seen this on. This is the second one used for ownCloud, and I have another that is simply a FTP server for some old backups that has done this in the past.

                    All three of the above systems are at different clients sites. with different servers and networks. All three are on Hyper-V though.
                    Those two systems have not done this recently, but this system is doing it daily now.

                    Is your DHCP server on the same Hyper-V server?

                    Also, what kind of network switch is it connected to?

                    What part of happens on three different networks did you not grasp?

                    dafyreD 1 Reply Last reply Reply Quote 0
                    • dafyreD
                      dafyre @JaredBusch
                      last edited by

                      @JaredBusch said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

                      @dafyre said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

                      @JaredBusch said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

                      @dafyre said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

                      Do you have DHCP server set to give this server a specific IP?

                      It is a DHCP reservation yes.

                      I could obviously make this static, but i hate static, because what if I change the network scope or DNS or something. Too much to manually update in that instance.

                      Also this is NOT the first CentOS 7 system I have seen this on. This is the second one used for ownCloud, and I have another that is simply a FTP server for some old backups that has done this in the past.

                      All three of the above systems are at different clients sites. with different servers and networks. All three are on Hyper-V though.
                      Those two systems have not done this recently, but this system is doing it daily now.

                      Is your DHCP server on the same Hyper-V server?

                      Also, what kind of network switch is it connected to?

                      What part of happens on three different networks did you not grasp?

                      In one eye, out the other, I reckon.

                      What does your DHCP server show when you restart the network services on the ownCloud box?

                      JaredBuschJ 1 Reply Last reply Reply Quote 0
                      • JaredBuschJ
                        JaredBusch @dafyre
                        last edited by JaredBusch

                        @dafyre haven't looked. when i restart i get the address. so i would assume it is all good. I guess the server log can be checked for the time of failure though.

                        1 Reply Last reply Reply Quote 1
                        • momurdaM
                          momurda
                          last edited by

                          Is this maybe something that has happened over time to these servers as a result of updates? Nowadays the network interfaces are using a different naming scheme, starting in systemd v197. The naming of the interfaces are usually enox ensx, others unless disabled by various methods, ethx names are last in line to be used.

                          https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/

                          dafyreD 1 Reply Last reply Reply Quote 0
                          • dafyreD
                            dafyre @momurda
                            last edited by

                            @momurda said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

                            Is this maybe something that has happened over time to these servers as a result of updates? Nowadays the network interfaces are using a different naming scheme, starting in systemd v197. The naming of the interfaces are usually enox ensx, others unless disabled by various methods, ethx names are last in line to be used.

                            https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/

                            @JaredBusch fixed his interface name to always use eth0 on this server.

                            1 Reply Last reply Reply Quote 0
                            • JaredBuschJ
                              JaredBusch
                              last edited by

                              This is still a problem everywhere that I have Hyper-V Server 2012 R2 or Server 2012 R2 + Hyper-V.

                              It is very intermittent and It has been easier to just pop in and kick the service.
                              But now I have the time to handle this and I need to get this figured out.

                              Today I have a specific VM down that also went down yesterday. There are 2 other CentOS 7 VMs on this same server that are not down right now.

                              This VM went down yesterday also.

                              1 Reply Last reply Reply Quote 0
                              • JaredBuschJ
                                JaredBusch
                                last edited by

                                0_1485362583958_upload-109b49db-aa05-47e4-a71e-19e5b0efdedb

                                The ownCloud instance is the one in quesiton today. It was rebooted yesterday because after getting alerted to it being down, I also updated ownCloud from 9.0.x to 9.1.x and then gave it a reboot for good measure.

                                As you can see everything else was rebooted 2 days ago. Because I rebooted the entire hypervisor (Hyper-V Server 2012 R2) on Sunday for updates.

                                0_1485362842946_upload-f3743a3d-9a58-4124-8a28-925b43eed3b1

                                This is the entire DHCP lease list. You can see it is not a large site.
                                0_1485363143422_upload-8ca29cad-2f05-44e0-8ffe-9847973265e1

                                1 Reply Last reply Reply Quote 0
                                • JaredBuschJ
                                  JaredBusch
                                  last edited by JaredBusch

                                  And the same problem that I had before. Bad NTP.

                                  0_1485363690487_upload-84fba01a-de24-4f1c-b510-2caceca2b190

                                  But this time I am only using two internal sources. There is not a third internal source available at this site to have a proper witness.

                                  1 Reply Last reply Reply Quote 0
                                  • JaredBuschJ
                                    JaredBusch
                                    last edited by

                                    Edited chrony to use 3 external servers and rebooted..

                                    # These servers were defined in the installation:
# server 10.201.1.7 iburst
# server 10.201.1.1 iburst
server 0.us.pool.ntp.org iburst
server 1.us.pool.ntp.org iburst
server 2.us.pool.ntp.org iburst
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).

                                    /var/log/messages:Jan 26 11:55:07 owncloud chronyd[617]: chronyd version 2.1.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP $
/var/log/messages:Jan 26 11:55:07 owncloud chronyd[617]: Frequency -22.354 +/- 0.114 ppm read from /var/lib/chrony/drift
/var/log/messages:Jan 25 11:06:29 owncloud chronyd[617]: Selected source 108.61.73.244
/var/log/messages:Jan 25 11:06:29 owncloud chronyd[617]: System clock wrong by -89327.780923 seconds, adjustment started
/var/log/messages:Jan 25 11:06:29 owncloud chronyd[617]: System clock was stepped by -89327.780923 seconds
                                    1 Reply Last reply Reply Quote 0
                                    • JaredBuschJ
                                      JaredBusch
                                      last edited by

                                      Seriously, are there issues with Chrony?

                                      1 Reply Last reply Reply Quote 0
                                      • JaredBuschJ
                                        JaredBusch
                                        last edited by gjacobse

                                        Server a spun up a month ago for community.bundystl.com just was unreachable.

                                        Guess what.. WTF 4 active damned servers listed... why can it not stay f***ing working?

                                        [root@bna-cmty ~]# nano /etc/chrony.conf
# These servers were defined in the installation:
server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).

                                        [root@bna-cmty ~]# grep chrony /var/log/messages*
/var/log/messages:Jan 22 20:15:23 bna-cmty chronyd[621]: Selected source 74.120.81.219
/var/log/messages:Feb 14 02:13:15 bna-cmty chronyd[618]: chronyd version 2.1.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +DEBUG +ASYNCDNS +IPV6 +SECHASH)
/var/log/messages:Feb 14 02:13:15 bna-cmty chronyd[618]: Frequency -18.365 +/- 0.044 ppm read from /var/lib/chrony/drift
/var/log/messages:Jan 25 21:01:26 bna-cmty chronyd[618]: Selected source 104.232.3.3
/var/log/messages:Jan 25 21:01:26 bna-cmty chronyd[618]: System clock wrong by -1660320.927108 seconds, adjustment started
/var/log/messages:Jan 25 21:01:26 bna-cmty chronyd[618]: System clock was stepped by -1660320.927108 seconds
/var/log/messages:Jan 25 21:02:31 bna-cmty chronyd[618]: Selected source 129.250.35.251
/var/log/messages:Jan 26 05:29:06 bna-cmty chronyd[618]: Can't synchronise: no selectable sources
/var/log/messages:Jan 28 00:09:15 bna-cmty chronyd[618]: Source 104.232.3.3 offline
/var/log/messages:Jan 28 00:09:15 bna-cmty chronyd[618]: Source 129.250.35.251 offline
/var/log/messages:Jan 28 00:09:15 bna-cmty chronyd[618]: Source 104.245.33.76 offline
/var/log/messages:Jan 28 00:09:15 bna-cmty chronyd[618]: Source 66.228.42.59 offline
/var/log/messages:Jan 28 00:09:15 bna-cmty chronyd[618]: Source 104.232.3.3 online
/var/log/messages:Jan 28 00:09:15 bna-cmty chronyd[618]: Source 129.250.35.251 online
/var/log/messages:Jan 28 00:09:15 bna-cmty chronyd[618]: Source 104.245.33.76 online
/var/log/messages:Jan 28 00:09:15 bna-cmty chronyd[618]: Source 66.228.42.59 online
/var/log/messages:Jan 28 00:09:16 bna-cmty chronyd[618]: Selected source 129.250.35.251
/var/log/messages-20170101:Dec 25 23:04:57 bna-cmty yum[37947]: Updated: chrony-2.1.1-4.el7.centos.x86_64
/var/log/messages-20170101:Dec 25 23:06:33 bna-cmty chronyd[736]: chronyd exiting
/var/log/messages-20170101:Dec 25 23:06:33 bna-cmty chronyd[4840]: chronyd version 2.1.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +DEBUG +ASYNCDNS +IPV6 +SECHASH)
/var/log/messages-20170101:Dec 25 23:06:33 bna-cmty chronyd[4840]: Frequency -17.944 +/- 0.124 ppm read from /var/lib/chrony/drift
/var/log/messages-20170101:Dec 25 23:06:38 bna-cmty chronyd[4840]: Selected source 192.155.90.13
/var/log/messages-20170101:Jan 19 00:13:55 bna-cmty chronyd[638]: chronyd version 2.1.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +DEBUG +ASYNCDNS +IPV6 +SECHASH)
/var/log/messages-20170101:Jan 19 00:13:56 bna-cmty chronyd[638]: Frequency -17.944 +/- 0.136 ppm read from /var/lib/chrony/drift
/var/log/messages-20170101:Dec 26 00:39:42 bna-cmty chronyd[638]: Selected source 209.50.48.237
/var/log/messages-20170101:Dec 26 00:39:42 bna-cmty chronyd[638]: System clock wrong by -2072062.504299 seconds, adjustment started
/var/log/messages-20170101:Dec 26 00:39:42 bna-cmty chronyd[638]: System clock was stepped by -2072062.504299 seconds
/var/log/messages-20170101:Dec 26 02:37:08 bna-cmty chronyd[638]: Selected source 66.151.147.38
/var/log/messages-20170101:Dec 26 09:17:31 bna-cmty chronyd[638]: Can't synchronise: no selectable sources
/var/log/messages-20170101:Dec 26 15:03:37 bna-cmty chronyd[638]: Source 74.120.8.2 offline
/var/log/messages-20170101:Dec 26 15:03:37 bna-cmty chronyd[638]: Source 66.151.147.38 offline
/var/log/messages-20170101:Dec 26 15:03:37 bna-cmty chronyd[638]: Source 66.135.44.92 offline
/var/log/messages-20170101:Dec 26 15:03:37 bna-cmty chronyd[638]: Source 209.50.48.237 offline
/var/log/messages-20170101:Dec 26 15:03:38 bna-cmty chronyd[638]: Source 74.120.8.2 online
/var/log/messages-20170101:Dec 26 15:03:38 bna-cmty chronyd[638]: Source 66.151.147.38 online
/var/log/messages-20170101:Dec 26 15:03:38 bna-cmty chronyd[638]: Source 66.135.44.92 online
/var/log/messages-20170101:Dec 26 15:03:38 bna-cmty chronyd[638]: Source 209.50.48.237 online
/var/log/messages-20170101:Dec 26 15:03:38 bna-cmty chronyd[638]: Selected source 209.50.48.237
/var/log/messages-20170101:Dec 26 18:04:47 bna-cmty chronyd[638]: Selected source 66.151.147.38
/var/log/messages-20170101:Dec 27 17:55:13 bna-cmty chronyd[638]: Selected source 209.50.48.237
/var/log/messages-20170101:Dec 28 07:08:18 bna-cmty chronyd[638]: Selected source 66.151.147.38
/var/log/messages-20170101:Dec 28 11:35:31 bna-cmty chronyd[638]: Selected source 209.50.48.237
/var/log/messages-20170101:Dec 28 18:03:54 bna-cmty chronyd[638]: Selected source 66.151.147.38
/var/log/messages-20170101:Dec 30 20:46:01 bna-cmty chronyd[638]: Selected source 209.50.48.237
/var/log/messages-20170101:Dec 31 04:48:48 bna-cmty chronyd[638]: Selected source 66.151.147.38
/var/log/messages-20170101:Dec 31 13:00:24 bna-cmty chronyd[638]: System clock status set to insert leap second
/var/log/messages-20170101:Dec 31 18:00:01 bna-cmty chronyd[638]: System clock status set to not insert/delete leap second
/var/log/messages-20170101:Dec 31 18:01:48 bna-cmty chronyd[638]: Source 74.120.8.2 replaced with 69.89.207.99
/var/log/messages-20170101:Dec 31 18:08:16 bna-cmty chronyd[638]: Selected source 69.89.207.99
/var/log/messages-20170109:Jan 18 06:33:17 bna-cmty chronyd[621]: chronyd version 2.1.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +DEBUG +ASYNCDNS +IPV6 +SECHASH)
/var/log/messages-20170109:Jan 18 06:33:17 bna-cmty chronyd[621]: Frequency -18.334 +/- 0.010 ppm read from /var/lib/chrony/drift
/var/log/messages-20170109:Jan  6 15:37:26 bna-cmty chronyd[621]: Selected source 69.89.207.99
/var/log/messages-20170109:Jan  6 15:37:26 bna-cmty chronyd[621]: System clock wrong by -1004160.299142 seconds, adjustment started
/var/log/messages-20170109:Jan  6 15:37:26 bna-cmty chronyd[621]: System clock was stepped by -1004160.299142 seconds
/var/log/messages-20170109:Jan  6 22:08:29 bna-cmty chronyd[621]: Source 198.55.111.50 offline
/var/log/messages-20170109:Jan  6 22:08:29 bna-cmty chronyd[621]: Source 204.2.134.162 offline
/var/log/messages-20170109:Jan  6 22:08:29 bna-cmty chronyd[621]: Source 74.120.81.219 offline
/var/log/messages-20170109:Jan  6 22:08:29 bna-cmty chronyd[621]: Source 69.89.207.99 offline
/var/log/messages-20170109:Jan  6 22:08:29 bna-cmty chronyd[621]: Can't synchronise: no selectable sources
/var/log/messages-20170109:Jan  6 22:08:30 bna-cmty chronyd[621]: Source 198.55.111.50 online
/var/log/messages-20170109:Jan  6 22:08:30 bna-cmty chronyd[621]: Source 204.2.134.162 online
/var/log/messages-20170109:Jan  6 22:08:30 bna-cmty chronyd[621]: Source 69.89.207.99 online
/var/log/messages-20170109:Jan  6 22:08:30 bna-cmty chronyd[621]: Source 74.120.81.219 online
/var/log/messages-20170109:Jan  6 22:08:30 bna-cmty chronyd[621]: Selected source 69.89.207.99
                                        1 Reply Last reply Reply Quote 0
                                        • ObsolesceO
                                          Obsolesce
                                          last edited by

                                          Is there a specific reason I missed why you arent using the vm integration service time sync with the host?

                                          Could you install the latest hyperV LIS on the linux vms, enable the guest time sync, and use windows/microsoft time sync on the host if its not on the domain?

                                          1 Reply Last reply Reply Quote 0
                                          • stacksofplatesS
                                            stacksofplates
                                            last edited by stacksofplates

                                            I'm amazed that just restarting the service fixes the issue. I wonder if the hardware clock isn't being updated and the skew is too great so Chrony just gives up.

                                            If you install ntpdate and enable it, it will update the hw clock every 11 minutes. I don't think it's installed by default with the minimal installs.

                                            You could also just go back to ntpd. It's still available and might not have the issue. It also has a file under /etc/sysconfig/ntpd where you can set ntpd to update the hw clock.

                                            JaredBuschJ 1 Reply Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 3
                                            • 3 / 3
                                            • First post
                                              Last post