Deleting eth1 deleted eth0 also; rendering the guest OS completely unusable!

bislinks's picture
Submitted by bislinks on Mon, 12/18/2017 - 08:16 Pro Licensee

On a particular Virtualmin/Cloudmin machine (ns83 KVM Host System) which hosts/controls several guest/physical OSs, I was having trouble with a particular guest OS ( git3 ) for quite a while. It has 100 GB virtual HDD, 4 GB RAM, and 2 virtual processors.

It had two real interfaces eth0 and eth1 and had local IPs assigned to them and was working fine. My plan was to delete the second interface and assign a public IP to the first one. But as soon as I deleted eth1, it also deleted eth0 rendering the guest OS unusable. When the instance is down, in the network interfaces, it does not give me a chance to edit eth0 at all. So there is no way to edit this interface.

Even after restarting the guest OS a number times and even restarting the physical machine that hosts this guest, this guest OS is still not starting

The error it gives is this:

started, but a problem was detected : KVM instance was started, but could not be pinged after 60 seconds

But in reality, the KVM instance is not started at all even after repeated attempts!

Re-fetching system status ..
.. done. New status is : Down - KVM instance is down

There are other OSs ( guests ns45 and ns47 and physical servers ) controlled by this physical server (Intel Xeon 8 cores, 30 GB RAM, 900 GB HDD), which are doing just fine.

Also tried creating a clone of this guest; but the clone does the same thing as the original OS.

Files: 
Image icon screenshot_from_2017-12-18_08-04-47.png
Image icon screenshot_from_2017-12-19_06-34-00.png
Image icon screenshot_from_2018-01-10_10-24-08.png
Status: 
Active

Comments

Submitted by JamieCameron on Tue, 12/19/2017 - 01:15

When the VM is down, can you go to the System Configuration -> Network Interfaces page, and if so does eth0 or eth1 appear? If not, can you add eth0 back on that page?

bislinks's picture
Submitted by bislinks on Tue, 12/19/2017 - 06:34 Pro Licensee

eth0 and eth1 both can be seen. But no link to edit eth0.

Here is a snap shot:

Submitted by JamieCameron on Tue, 12/19/2017 - 19:18

Looks like the network config file that Virtualmin expects to be on the VM is missing. Which linux distribution is this VM running?

If it's debian or ubuntu, the file is /etc/network/interfaces

Submitted by JamieCameron on Fri, 12/29/2017 - 23:18

In that case, the file is /etc/sysconfig/network-scripts/ifcfg-eth0

bislinks's picture
Submitted by bislinks on Mon, 01/08/2018 - 10:40 Pro Licensee

The VM is not booting.

How do I add those missing files?

Submitted by JamieCameron on Tue, 01/09/2018 - 00:23

You can access the VM filesystem by SSHing into the Cloudmin master as root and running :

cloudmin mount-system --host your.vm.name

the FS will then be accessible under a directory on the VM host. When done, you need to run :

cloudmin umount-system --host your.vm.name

bislinks's picture
Submitted by bislinks on Wed, 01/10/2018 - 10:23 Pro Licensee

ifcfg-eth0 already exists in the VM in question:

[root@ns83 ~]# cloudmin mount-system --host 192.168.1.91
Mounting filesystem for 192.168.1.91 ..
.. mounted at /mnt/kvm-192.168.1.91 on host ns83.datalawn.com

[root@ns83 ~]# ls /mnt/kvm-192.168.1.91/etc/sysconfig/network-scripts/
ifcfg-eth0      ifdown       ifdown-ipv6    ifdown-sit       ifup-aliases  ifup-ipv6   ifup-ppp       ifup-tunnel
ifcfg-eth0.bak  ifdown-bnep  ifdown-isdn    ifdown-Team      ifup-bnep     ifup-isdn   ifup-routes    ifup-wireless
ifcfg-eth1.bak  ifdown-eth   ifdown-post    ifdown-TeamPort  ifup-eth      ifup-plip   ifup-sit       init.ipv6-global
ifcfg-eth2      ifdown-ib    ifdown-ppp     ifdown-tunnel    ifup-ib       ifup-plusb  ifup-Team      network-functions
ifcfg-lo        ifdown-ippp  ifdown-routes  ifup             ifup-ippp     ifup-post   ifup-TeamPort  network-functions-ipv6

[root@ns83 ~]# cat /mnt/kvm-192.168.1.91/etc/sysconfig/network-scripts/ifcfg-eth0
GATEWAY=192.168.1.254
NAME=""
BOOTPROTO=dhcp
MACADDR=""
IPV6INIT=yes
TYPE=Ethernet
DEVICE=eth0
MTU=""
ONBOOT=yes
ZONE=public
[root@ns83 ~]#

Also, for some reason, 192.168.1.4 stopped working suddenty with error: no route to host

On this particular Machine (ns83), only VMs with public IPs are working, none with 192.168.1.* are working at this moment.

bislinks's picture
Submitted by bislinks on Wed, 01/10/2018 - 10:25 Pro Licensee

Screen shot of all VMs on ns83

Submitted by JamieCameron on Thu, 01/11/2018 - 00:03

Odd that these VMs have three network interfaces - is that intentional? I'd be interested to see the contents of those ifcfg-eth0 and ifcfg-eth1 files.

bislinks's picture
Submitted by bislinks on Thu, 01/11/2018 - 09:30 Pro Licensee

I changed the hostname from 192.168.1.91 to 192.168.1.2 and deleted eth1 and eth2

Content of eth0:

[root@ns83 ~]# cloudmin mount-system --host 192.168.1.2
Mounting filesystem for 192.168.1.2 ..
.. mounted at /mnt/kvm-192.168.1.2 on host ns83.datalawn.com

[root@ns83 ~]# ls /mnt/kvm-192.168.1.2
aquota.group  bin   dev  home  lib64       media  opt   root  sbin  sys  usr
aquota.user   boot  etc  lib   lost+found  mnt    proc  run   srv   tmp  var

[root@ns83 ~]# ls /mnt/kvm-192.168.1.2/etc/sysconfig/network-scripts
ifcfg-eth0      ifdown-bnep  ifdown-isdn    ifdown-Team      ifup-bnep  ifup-isdn   ifup-routes    ifup-wireless
ifcfg-eth0.bak  ifdown-eth   ifdown-post    ifdown-TeamPort  ifup-eth   ifup-plip   ifup-sit       init.ipv6-global
ifcfg-eth1.bak  ifdown-ib    ifdown-ppp     ifdown-tunnel    ifup-ib    ifup-plusb  ifup-Team      network-functions
ifcfg-lo        ifdown-ippp  ifdown-routes  ifup             ifup-ippp  ifup-post   ifup-TeamPort  network-functions-ipv6
ifdown          ifdown-ipv6  ifdown-sit     ifup-aliases     ifup-ipv6  ifup-ppp    ifup-tunnel

[root@ns83 ~]# cat /mnt/kvm-192.168.1.2/etc/sysconfig/network-scripts/ifcfg-eth0
BOOTPROTO=dhcp
NAME=""
GATEWAY=192.168.1.254
MACADDR=""
TYPE=Ethernet
IPV6INIT=yes
DEVICE=eth0
MTU=""
ONBOOT=yes
ZONE=public
[root@ns83 ~]#

bislinks's picture
Submitted by bislinks on Sun, 01/14/2018 - 08:24 Pro Licensee

Network Details that might be helpful:

[root@ns83 ~]# ifconfig |more
br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 76.209.12.3  netmask 255.255.255.192  broadcast 76.209.12.63
        inet6 fe80::222:19ff:fec4:fb25  prefixlen 64  scopeid 0x20<link>
        inet6 2600:1700:89e0:5ea0:222:19ff:fec4:fb25  prefixlen 64  scopeid 0x0<global>
        ether 00:22:19:c4:fb:25  txqueuelen 1000  (Ethernet)
        RX packets 6794426  bytes 3540047744 (3.2 GiB)
        RX errors 0  dropped 296461  overruns 0  frame 0
        TX packets 4334894  bytes 4794536234 (4.4 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        ether 02:42:5c:c0:8d:96  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::222:19ff:fec4:fb25  prefixlen 64  scopeid 0x20<link>
        ether 00:22:19:c4:fb:25  txqueuelen 1000  (Ethernet)
        RX packets 8002477  bytes 3826618143 (3.5 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5396637  bytes 4816235044 (4.4 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eno2: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 00:22:19:c4:fb:27  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 7106110  bytes 10963799217 (10.2 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7106110  bytes 10963799217 (10.2 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::5847:ceff:fe7e:71cd  prefixlen 64  scopeid 0x20<link>
        ether 5a:47:ce:7e:71:cd  txqueuelen 1000  (Ethernet)
        RX packets 784362  bytes 80457431 (76.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1974471  bytes 167971495 (160.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::70de:a8ff:fe11:9544  prefixlen 64  scopeid 0x20<link>
        ether 72:de:a8:11:95:44  txqueuelen 1000  (Ethernet)
        RX packets 35405  bytes 3662245 (3.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1647960  bytes 119563161 (114.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::4ca2:16ff:fe41:8907  prefixlen 64  scopeid 0x20<link>
        ether 4e:a2:16:41:89:07  txqueuelen 1000  (Ethernet)
        RX packets 664556  bytes 78148596 (74.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2154476  bytes 195895819 (186.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:2e:bc:0b  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0

[root@ns83 ~]# hostname -f
ns83.datalawn.com

[root@ns83 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
76.209.12.3 ns83.datalawn.com

[root@ns83 ~]# cat /etc/hostname
ns83.datalawn.com

[root@ns83 ~]# cat /etc/resolv.conf
nameserver 127.0.0.1
nameserver 68.94.156.1
nameserver 192.168.1.254
search attlocal.net datalawn.com
; generated by /usr/sbin/dhclient-script

[root@ns83 ~]# cat /etc/sysconfig/network-scripts/ifcfg-br0
BOOTPROTO=none
NAME=""
GATEWAY=76.209.12.62
MACADDR=""
NM_CONTROLLED=no
IPV6INIT=yes
TYPE=Bridge
DEVICE=br0
MTU=""
NETMASK=255.255.255.192
BROADCAST=76.209.12.63
IPADDR=76.209.12.3
NETWORK=""
ONBOOT=yes
ZONE=public

[root@ns83 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno1
PEERDNS=yes
HWADDR=00:22:19:C4:FB:25
IPV6INIT=yes
IPV6_PEERROUTES=yes
BROADCAST=""
PEERROUTES=no
UUID=015037ef-2ad8-49d7-982a-a162aa389939
BOOTPROTO=none
NAME=""
MACADDR=""
TYPE=Ethernet
IPV6_FAILURE_FATAL=no
DEVICE=eno1
BRIDGE=br0
IPV6_AUTOCONF=yes
NETMASK=""
MTU=""
IPV6_DEFROUTE=yes
DEFROUTE=yes
IPADDR=""
NETWORK=""
IPV4_FAILURE_FATAL=no
ONBOOT=yes
ZONE=public
IPV6_PEERDNS=yes
[root@ns83 ~]#

bislinks's picture
Submitted by bislinks on Thu, 01/18/2018 - 18:55 Pro Licensee

When i startup a system, it says it has been started, but in reality it is not!

Strange things happening on this server!

Submitted by JamieCameron on Sun, 01/21/2018 - 18:00

On a VM (which is the case here), the "physical" NIC can be removed by changing the VM's configuration.

@bislinks - are you using network bridges on your VM?

bislinks's picture
Submitted by bislinks on Sun, 01/21/2018 - 21:34 Pro Licensee

@JamieCameron Yes, br0, on ns83, the Metal Server that hosts this Guest VM.

bislinks's picture
Submitted by bislinks on Thu, 02/22/2018 - 12:22 Pro Licensee

After removing docker, the guest VMs are functioning properly!!!!

Submitted by JamieCameron on Sat, 02/24/2018 - 19:47

Wow .. maybe docker is also doing something with bridged interfaces that is interfering with KVM?