Nodes not starting [EVE-NG Pro, running on Unraid/KVM)
Posted: Sat Apr 15, 2023 1:45 am
Code: Select all
Linux eve-ng 4.20.17.2-eve-ng-uksm-wg+ #1 SMP Sat Feb 6 11:21:53 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux- Open existing topology that worked a month ago (VM has been off)
- Start node, it goes blue, shows success in top right corner
- Open console, nothing there (Native or HTML5)
Host System: Dual Xeon 16 thread (32 total), 128GB RAM
Host OS: Unraid, EVE-NG Pro running in KVM (cpu-passthrough). VM has 30 CPU cores, 96GiB RAM
License good, time/date good.
VT-X/VT-D all good on host and EVE-NG VM
Permissions fixed, paths look good.
Node in this case is Juniper vSRX-NG, seems to happen for all my nodes.
Node is running with "-machine type=pc,accel=kvm -cpu host -serial mon:stdio -nographic -no-user-config -nodefaults -rtc base=utc" - this worked previously.
Log extract - /opt/unetlab/data/Logs/unl_wrapper.txt
Code: Select all
Apr 15 13:38:34 Apr 15 13:38:35 Online Check state: Valid
Apr 15 13:38:35 Apr 15 13:38:35 ERROR: sat=0
Apr 15 13:38:35 Apr 15 13:38:35 LOG: will start node 16 0 300 1 44141
Apr 15 13:38:35 INFO: tunctl -u 33808 -g root -t vun001000001000 2>&1
Apr 15 13:38:35 INFO: sysctl -w net.ipv6.conf.vun001000001000.disable_ipv6=1
Apr 15 13:38:35 INFO: vun001000001000 -> 0,0,0,0
Apr 15 13:38:35 LOG: exec : brctl addif internal_1 vun001000001000 2>&1
sh: 1: ovs-vsctl: not found
Apr 15 13:38:35 INFO: tunctl -u 33808 -g root -t vun001000001001 2>&1
Apr 15 13:38:35 INFO: sysctl -w net.ipv6.conf.vun001000001001.disable_ipv6=1
Apr 15 13:38:35 INFO: vun001000001001 -> 0,0,0,0
Apr 15 13:38:35 INFO: tunctl -u 33808 -g root -t vun001000001002 2>&1
Apr 15 13:38:35 INFO: sysctl -w net.ipv6.conf.vun001000001002.disable_ipv6=1
Apr 15 13:38:35 INFO: vun001000001002 -> 0,0,0,0
Apr 15 13:38:35 LOG: exec : sysctl -w net.ipv6.conf.vnet00100000001.disable_ipv6=1
Apr 15 13:38:35 LOG: exec : sysctl -w net.ipv6.conf.vun001000001002.disable_ipv6=1
Apr 15 13:38:35 LOG: exec : brctl addif vnet00100000001 vun001000001002 2>&1
sh: 1: ovs-vsctl: not found
Apr 15 13:38:35 INFO: tunctl -u 33808 -g root -t vun001000001003 2>&1
Apr 15 13:38:35 INFO: sysctl -w net.ipv6.conf.vun001000001003.disable_ipv6=1
Apr 15 13:38:35 INFO: vun001000001003 -> 0,0,0,0
Apr 15 13:38:35 LOG: exec : sysctl -w net.ipv6.conf.vnet00100000012.disable_ipv6=1
Apr 15 13:38:35 LOG: exec : sysctl -w net.ipv6.conf.vun001000001003.disable_ipv6=1
Apr 15 13:38:35 LOG: exec : brctl addif vnet00100000012 vun001000001003 2>&1
sh: 1: ovs-vsctl: not found
Apr 15 13:38:35 INFO: tunctl -u 33808 -g root -t vun001000001004 2>&1
Apr 15 13:38:35 INFO: sysctl -w net.ipv6.conf.vun001000001004.disable_ipv6=1
Apr 15 13:38:35 INFO: vun001000001004 -> 0,0,0,0
Apr 15 13:38:35 LOG: exec : sysctl -w net.ipv6.conf.vnet00100000006.disable_ipv6=1
Apr 15 13:38:35 LOG: exec : sysctl -w net.ipv6.conf.vun001000001004.disable_ipv6=1
Apr 15 13:38:35 LOG: exec : brctl addif vnet00100000006 vun001000001004 2>&1
sh: 1: ovs-vsctl: not found
Apr 15 13:38:35 INFO: tunctl -u 33808 -g root -t vun001000001005 2>&1
Apr 15 13:38:35 INFO: sysctl -w net.ipv6.conf.vun001000001005.disable_ipv6=1
Apr 15 13:38:35 INFO: vun001000001005 -> 0,0,0,0
Apr 15 13:38:35 INFO: tunctl -u 33808 -g root -t vun001000001006 2>&1
Apr 15 13:38:35 INFO: sysctl -w net.ipv6.conf.vun001000001006.disable_ipv6=1
Apr 15 13:38:35 INFO: vun001000001006 -> 0,0,0,0
Apr 15 13:38:35 INFO: tunctl -u 33808 -g root -t vun001000001007 2>&1
Apr 15 13:38:35 INFO: sysctl -w net.ipv6.conf.vun001000001007.disable_ipv6=1
Apr 15 13:38:35 INFO: vun001000001007 -> 0,0,0,0
Apr 15 13:38:36 INFO: tunctl -u 33808 -g root -t vun001000001008 2>&1
Apr 15 13:38:36 INFO: sysctl -w net.ipv6.conf.vun001000001008.disable_ipv6=1
Apr 15 13:38:36 INFO: vun001000001008 -> 0,0,0,0
Apr 15 13:38:36 INFO: tunctl -u 33808 -g root -t vun001000001009 2>&1
Apr 15 13:38:36 INFO: sysctl -w net.ipv6.conf.vun001000001009.disable_ipv6=1
Apr 15 13:38:36 INFO: vun001000001009 -> 0,0,0,0
Apr 15 13:38:36 ERROR: QEMU Arch is not set (80015).
Apr 15 13:38:36 INFO: CWD is /opt/unetlab/tmp/0/eff7d377-147f-4fd5-8f50-56cdf19f377d/16
Apr 15 13:38:36 INFO: starting /opt/unetlab/wrappers/qemu_wrapper -C 44141 -T 0 -D 16 -t "CE2" -F /opt/qemu-4.1.0/bin/qemu-system-x86_64 -d 0 -- -nographic -device virtio-net-pci,netdev=net0,mac=50:00:00:10:00:00 -netdev tap,id=net0,ifname=vun001000001000,script=no -device virtio-net-pci,netdev=net1,mac=50:00:00:10:00:01 -netdev tap,id=net1,ifname=vun001000001001,script=no -device virtio-net-pci,netdev=net2,mac=50:00:00:10:00:02 -netdev tap,id=net2,ifname=vun001000001002,script=no -device virtio-net-pci,netdev=net3,mac=50:00:00:10:00:03 -netdev tap,id=net3,ifname=vun001000001003,script=no -device virtio-net-pci,netdev=net4,mac=50:00:00:10:00:04 -netdev tap,id=net4,ifname=vun001000001004,script=no -device virtio-net-pci,netdev=net5,mac=50:00:00:10:00:05 -netdev tap,id=net5,ifname=vun001000001005,script=no -device virtio-net-pci,netdev=net6,mac=50:00:00:10:00:06 -netdev tap,id=net6,ifname=vun001000001006,script=no -device virtio-net-pci,netdev=net7,mac=50:00:00:10:00:07 -netdev tap,id=net7,ifname=vun001000001007,script=no -device virtio-net-pci,netdev=net8,mac=50:00:00:10:00:08 -netdev tap,id=net8,ifname=vun001000001008,script=no -device virtio-net-pci,netdev=net9,mac=50:00:00:10:00:09 -netdev tap,id=net9,ifname=vun001000001009,script=no -smp 2 -m 4096 -name CE2 -uuid b705159a-0657-4f1f-ac86-84a5ce6ae4d8 -qmp unix:./qmp-sock,server,nowait -monitor unix:./mon-sock,server,nowait -monitor unix:./mon2-sock,server,nowait -drive file=virtioa.qcow2,if=virtio,bus=0,unit=0,cache=none -machine type=pc,accel=kvm -cpu host -serial mon:stdio -nographic -no-user-config -nodefaults -rtc base=utc
Apr 15 13:38:36 INFO: starting sudo systemd-run -G --no-block --property=WorkingDirectory=/opt/unetlab/tmp/0/eff7d377-147f-4fd5-8f50-56cdf19f377d/16 --service-type=simple --uid=0 --gid=32768 --unit=eveng_0@eff7d377-147f-4fd5-8f50-56cdf19f377d@16 /opt/unetlab/wrappers/qemu_wrapper -C 44141 -T 0 -D 16 -t "CE2" -F /opt/qemu-4.1.0/bin/qemu-system-x86_64 -d 0 -- -nographic -device virtio-net-pci,netdev=net0,mac=50:00:00:10:00:00 -netdev tap,id=net0,ifname=vun001000001000,script=no -device virtio-net-pci,netdev=net1,mac=50:00:00:10:00:01 -netdev tap,id=net1,ifname=vun001000001001,script=no -device virtio-net-pci,netdev=net2,mac=50:00:00:10:00:02 -netdev tap,id=net2,ifname=vun001000001002,script=no -device virtio-net-pci,netdev=net3,mac=50:00:00:10:00:03 -netdev tap,id=net3,ifname=vun001000001003,script=no -device virtio-net-pci,netdev=net4,mac=50:00:00:10:00:04 -netdev tap,id=net4,ifname=vun001000001004,script=no -device virtio-net-pci,netdev=net5,mac=50:00:00:10:00:05 -netdev tap,id=net5,ifname=vun001000001005,script=no -device virtio-net-pci,netdev=net6,mac=50:00:00:10:00:06 -netdev tap,id=net6,ifname=vun001000001006,script=no -device virtio-net-pci,netdev=net7,mac=50:00:00:10:00:07 -netdev tap,id=net7,ifname=vun001000001007,script=no -device virtio-net-pci,netdev=net8,mac=50:00:00:10:00:08 -netdev tap,id=net8,ifname=vun001000001008,script=no -device virtio-net-pci,netdev=net9,mac=50:00:00:10:00:09 -netdev tap,id=net9,ifname=vun001000001009,script=no -smp 2 -m 4096 -name CE2 -uuid b705159a-0657-4f1f-ac86-84a5ce6ae4d8 -qmp unix:./qmp-sock,server,nowait -monitor unix:./mon-sock,server,nowait -monitor unix:./mon2-sock,server,nowait -drive file=virtioa.qcow2,if=virtio,bus=0,unit=0,cache=none -machine type=pc,accel=kvm -cpu host -serial mon:stdio -nographic -no-user-config -nodefaults -rtc base=utc
Running as unit: eveng_0@eff7d377-147f-4fd5-8f50-56cdf19f377d@16.service
Apr 15 13:38:36 INFO: starting systemctl show --property MainPID --value eveng_0@eff7d377-147f-4fd5-8f50-56cdf19f377d@16
Apr 15 13:38:36 INFO: started process is 31559
PHP Warning: unlink(/opt/unetlab/tmp/0/eff7d377-147f-4fd5-8f50-56cdf19f377d/16/l1down_0): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1146
Apr 15 13:38:36 INFO: vun001000001000 -> 0,0,0,0
PHP Warning: unlink(/opt/unetlab/tmp/0/eff7d377-147f-4fd5-8f50-56cdf19f377d/16/l1up_1): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1167
PHP Warning: unlink(/opt/unetlab/tmp/0/eff7d377-147f-4fd5-8f50-56cdf19f377d/16/suspend_1): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1169
PHP Warning: unlink(/opt/unetlab/tmp/0/eff7d377-147f-4fd5-8f50-56cdf19f377d/16/l1down_2): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1146
Apr 15 13:38:36 INFO: vun001000001002 -> 0,0,0,0
PHP Warning: unlink(/opt/unetlab/tmp/0/eff7d377-147f-4fd5-8f50-56cdf19f377d/16/l1down_3): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1146
Apr 15 13:38:36 INFO: vun001000001003 -> 0,0,0,0
PHP Warning: unlink(/opt/unetlab/tmp/0/eff7d377-147f-4fd5-8f50-56cdf19f377d/16/l1down_4): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1146
Apr 15 13:38:36 INFO: vun001000001004 -> 0,0,0,0
PHP Warning: unlink(/opt/unetlab/tmp/0/eff7d377-147f-4fd5-8f50-56cdf19f377d/16/l1up_5): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1167
PHP Warning: unlink(/opt/unetlab/tmp/0/eff7d377-147f-4fd5-8f50-56cdf19f377d/16/suspend_5): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1169
PHP Warning: unlink(/opt/unetlab/tmp/0/eff7d377-147f-4fd5-8f50-56cdf19f377d/16/l1up_6): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1167
PHP Warning: unlink(/opt/unetlab/tmp/0/eff7d377-147f-4fd5-8f50-56cdf19f377d/16/suspend_6): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1169
PHP Warning: unlink(/opt/unetlab/tmp/0/eff7d377-147f-4fd5-8f50-56cdf19f377d/16/l1up_7): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1167
PHP Warning: unlink(/opt/unetlab/tmp/0/eff7d377-147f-4fd5-8f50-56cdf19f377d/16/suspend_7): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1169
PHP Warning: unlink(/opt/unetlab/tmp/0/eff7d377-147f-4fd5-8f50-56cdf19f377d/16/l1up_8): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1167
PHP Warning: unlink(/opt/unetlab/tmp/0/eff7d377-147f-4fd5-8f50-56cdf19f377d/16/suspend_8): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1169
PHP Warning: unlink(/opt/unetlab/tmp/0/eff7d377-147f-4fd5-8f50-56cdf19f377d/16/l1up_9): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1167
PHP Warning: unlink(/opt/unetlab/tmp/0/eff7d377-147f-4fd5-8f50-56cdf19f377d/16/suspend_9): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1169
Apr 15 13:38:36 Apr 15 13:38:36 ERROR: node 16