first 3-4 days i can use my lab as usual, but after them i can't start new nodes or wipe/start existing running nodes.
Only server reboot may fix this, but after several days situation repeating.
Eve-ng installed on ESXi, one server, no memory/CPU limits are reached.
16 CPU/80GB RAM
EVE-NG 6.2.0-4-PRO
my /opt/unetlab/data/Logs/unl_wrapper.txt contains this when i trying to start node:
(after server rebooting log messages are the same, but node starting as normal)
Code: Select all
Jul 19 15:06:37 WARNING: Attribute ignored, invalid node_serial (40013).
Jul 19 15:06:37 WARNING: Attribute ignored, invalid node_serial (40013).
Jul 19 15:06:37 WARNING: Attribute ignored, invalid node_serial (40013).
Jul 19 15:06:37 WARNING: Attribute ignored, invalid node_serial (40013).
Jul 19 15:06:37 WARNING: Attribute ignored, invalid node_serial (40013).
Jul 19 15:06:37 WARNING: Attribute ignored, invalid node_serial (40013).
Jul 19 15:06:37 WARNING: Attribute ignored, invalid node_serial (40013).
Jul 19 15:06:37 WARNING: Attribute ignored, invalid node_serial (40013).
Jul 19 15:06:37 WARNING: Attribute ignored, invalid node_serial (40013).
Jul 19 15:06:37 WARNING: Attribute ignored, invalid node_serial (40013).
Jul 19 15:06:37 WARNING: Attribute ignored, invalid node_serial (40013).
Jul 19 15:06:37 WARNING: Attribute ignored, invalid node_serial (40013).
Jul 19 15:06:37 WARNING: Attribute ignored, invalid node_serial (40013).
Jul 19 15:06:37 Jul 19 15:06:37 Online Check state: Valid
Jul 19 15:06:38 Jul 19 15:06:38 ERROR: sat=0
Jul 19 15:06:38 Jul 19 15:06:38 DEBUG: vnet00100000008 cannot be smart
Jul 19 15:06:38 DEBUG: set vnet00100000004 0
Jul 19 15:06:38 DEBUG: set vnet00100000009 0
Jul 19 15:06:38 DEBUG: set vnet0010000001d 0
Jul 19 15:06:38 DEBUG: set vnet00100000014 0
Jul 19 15:06:38 Jul 19 15:06:38 LOG: will start node 12 0 300 1 51337
Jul 19 15:06:38 INFO: tunctl -u 33804 -g root -t vun001000000c00 2>&1
Jul 19 15:06:38 INFO: sysctl -w net.ipv6.conf.vun001000000c00.disable_ipv6=1
Jul 19 15:06:38 INFO: vun001000000c00 -> 0,0,0,0
PHP Warning: unlink(/opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/tc_0): No such file or directory in /opt/unetlab/html/includes/cli.php on line 1067
Jul 19 15:06:38 LOG: exec : brctl addif pnet1 vun001000000c00 2>&1
Jul 19 15:06:38 INFO: tunctl -u 33804 -g root -t vun001000000c01 2>&1
Jul 19 15:06:38 INFO: sysctl -w net.ipv6.conf.vun001000000c01.disable_ipv6=1
Jul 19 15:06:38 INFO: vun001000000c01 -> 0,0,0,0
PHP Warning: unlink(/opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/tc_1): No such file or directory in /opt/unetlab/html/includes/cli.php on line 1067
Jul 19 15:06:38 LOG: exec : sysctl -w net.ipv6.conf.vnet00100000004.disable_ipv6=1
Jul 19 15:06:38 LOG: exec : sysctl -w net.ipv6.conf.vun001000000c01.disable_ipv6=1
Jul 19 15:06:38 LOG: exec : brctl addif vnet00100000004 vun001000000c01 2>&1
Jul 19 15:06:38 DEBUG: set vun001000000c01 1
Jul 19 15:06:38 INFO: tunctl -u 33804 -g root -t vun001000000c02 2>&1
Jul 19 15:06:38 INFO: sysctl -w net.ipv6.conf.vun001000000c02.disable_ipv6=1
Jul 19 15:06:38 INFO: vun001000000c02 -> 0,0,0,0
PHP Warning: unlink(/opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/tc_2): No such file or directory in /opt/unetlab/html/includes/cli.php on line 1067
Jul 19 15:06:38 LOG: exec : sysctl -w net.ipv6.conf.vnet00100000009.disable_ipv6=1
Jul 19 15:06:38 LOG: exec : sysctl -w net.ipv6.conf.vun001000000c02.disable_ipv6=1
Jul 19 15:06:38 LOG: exec : brctl addif vnet00100000009 vun001000000c02 2>&1
Jul 19 15:06:38 DEBUG: set vun001000000c02 1
Jul 19 15:06:38 INFO: tunctl -u 33804 -g root -t vun001000000c03 2>&1
Jul 19 15:06:38 INFO: sysctl -w net.ipv6.conf.vun001000000c03.disable_ipv6=1
Jul 19 15:06:38 INFO: vun001000000c03 -> 0,0,0,0
PHP Warning: unlink(/opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/tc_3): No such file or directory in /opt/unetlab/html/includes/cli.php on line 1067
Jul 19 15:06:38 LOG: exec : sysctl -w net.ipv6.conf.vnet0010000001d.disable_ipv6=1
Jul 19 15:06:38 LOG: exec : sysctl -w net.ipv6.conf.vun001000000c03.disable_ipv6=1
Jul 19 15:06:38 LOG: exec : brctl addif vnet0010000001d vun001000000c03 2>&1
Jul 19 15:06:38 DEBUG: set vun001000000c03 1
Jul 19 15:06:38 INFO: tunctl -u 33804 -g root -t vun001000000c04 2>&1
Jul 19 15:06:38 INFO: sysctl -w net.ipv6.conf.vun001000000c04.disable_ipv6=1
Jul 19 15:06:38 INFO: vun001000000c04 -> 0,0,0,0
PHP Warning: unlink(/opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/tc_4): No such file or directory in /opt/unetlab/html/includes/cli.php on line 1067
Jul 19 15:06:38 LOG: exec : sysctl -w net.ipv6.conf.vnet00100000014.disable_ipv6=1
Jul 19 15:06:38 LOG: exec : sysctl -w net.ipv6.conf.vun001000000c04.disable_ipv6=1
Jul 19 15:06:38 LOG: exec : brctl addif vnet00100000014 vun001000000c04 2>&1
Jul 19 15:06:38 DEBUG: set vun001000000c04 1
Jul 19 15:06:38 INFO: tunctl -u 33804 -g root -t vun001000000c05 2>&1
Jul 19 15:06:38 INFO: sysctl -w net.ipv6.conf.vun001000000c05.disable_ipv6=1
Jul 19 15:06:38 INFO: vun001000000c05 -> 0,0,0,0
PHP Warning: unlink(/opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/tc_5): No such file or directory in /opt/unetlab/html/includes/cli.php on line 1067
Jul 19 15:06:38 INFO: tunctl -u 33804 -g root -t vun001000000c06 2>&1
Jul 19 15:06:38 INFO: sysctl -w net.ipv6.conf.vun001000000c06.disable_ipv6=1
Jul 19 15:06:38 INFO: vun001000000c06 -> 0,0,0,0
PHP Warning: unlink(/opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/tc_6): No such file or directory in /opt/unetlab/html/includes/cli.php on line 1067
Jul 19 15:06:39 INFO: tunctl -u 33804 -g root -t vun001000000c07 2>&1
Jul 19 15:06:39 INFO: sysctl -w net.ipv6.conf.vun001000000c07.disable_ipv6=1
Jul 19 15:06:39 INFO: vun001000000c07 -> 0,0,0,0
PHP Warning: unlink(/opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/tc_7): No such file or directory in /opt/unetlab/html/includes/cli.php on line 1067
Jul 19 15:06:39 ERROR: QEMU Arch is not set (80015).
Jul 19 15:06:39 cmd = mountpoint -q /opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/jail || mount -B -o ro /opt/unetlab/jail /opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/jail
Jul 19 15:06:39 cmd = mountpoint -q /opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/dev/bus/usb || mount -B /dev/bus/usb /opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/dev/bus/usb
Jul 19 15:06:39 cmd = mountpoint -q /opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/sys/bus/usb || mount -B -o ro /sys/bus/usb /opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/sys/bus/usb
Jul 19 15:06:39 cmd = mountpoint -q /opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/sys/devices || mount -B -o ro /sys/devices /opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/sys/devices
Jul 19 15:06:39 cmd = mountpoint -q /opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/sys/bus/usb || mount -B -o ro /sys/bus/usb /opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/sys/bus/usb
Jul 19 15:06:39 cmd = for i in /opt/qemu-*; do ( ls /opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/$i || mkdir /opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/$i) ; done
Jul 19 15:06:39 cmd = for i in /opt/qemu-*; do ( mountpoint -q /opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/$i || mount -B -o ro $i /opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/$i ) ; done
Jul 19 15:06:39 cmd = mountpoint -q /opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/opt/unetlab/addons/qemu || mount -B -o ro /opt/unetlab/addons/qemu /opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/opt/unetlab/addons/qemu
Jul 19 15:06:39 INFO: CWD is /opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12
Jul 19 15:06:39 INFO: starting /opt/unetlab/wrappers/qemu_wrapper -C 51337 -T 0 -D 12 -t "RT-156" -F /opt/qemu-4.1.0/bin/qemu-system-x86_64 -d 0 -- -nographic -device vmxnet3,netdev=net0,mac=50:00:00:0c:00:00 -netdev tap,id=net0,ifname=vun001000000c00,script=no -device vmxnet3,netdev=net1,mac=50:00:00:0c:00:01 -netdev tap,id=net1,ifname=vun001000000c01,script=no -device vmxnet3,netdev=net2,mac=50:00:00:0c:00:02 -netdev tap,id=net2,ifname=vun001000000c02,script=no -device vmxnet3,netdev=net3,mac=50:00:00:0c:00:03 -netdev tap,id=net3,ifname=vun001000000c03,script=no -device vmxnet3,netdev=net4,mac=50:00:00:0c:00:04 -netdev tap,id=net4,ifname=vun001000000c04,script=no -device vmxnet3,netdev=net5,mac=50:00:00:0c:00:05 -netdev tap,id=net5,ifname=vun001000000c05,script=no -device vmxnet3,netdev=net6,mac=50:00:00:0c:00:06 -netdev tap,id=net6,ifname=vun001000000c06,script=no -device vmxnet3,netdev=net7,mac=50:00:00:0c:00:07 -netdev tap,id=net7,ifname=vun001000000c07,script=no -smp 2 -m 4096 -name RT-156 -uuid 3d9ab490-dbe0-4559-a3a0-7afd4b574e58 -qmp unix:./qmp-sock,server,nowait -monitor unix:./mon-sock,server,nowait -monitor
unix:./mon2-sock,server,nowait -drive file=virtioa.qcow2,if=virtio,bus=0,unit=0,cache=none -machine type=pc,accel=kvm -cpu host -serial mon:stdio -nographic -no-user-config -nodefaults -rtc base=utc
Jul 19 15:06:39 INFO: starting systemd-run -G --no-block --property=ExecStopPost=/opt/unetlab/go/eve-clean --property=Environment=WDIR="/opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12" --property=Environment=TYPE="qemu" --property=WorkingDirectory=/opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12 --service-type=simple --uid=0 --gid=32768 --unit=eveng_0@5db226d0-ea3f-4830-86aa-401614192b5d@12 /opt/unetlab/wrappers/qemu_wrapper -C 51337 -T 0 -D 12 -t "RT-156" -F /opt/qemu-4.1.0/bin/qemu-system-x86_64 -d 0 -- -nographic -device vmxnet3,netdev=net0,mac=50:00:00:0c:00:00 -netdev tap,id=net0,ifname=vun001000000c00,script=no -device vmxnet3,netdev=net1,mac=50:00:00:0c:00:01 -netdev tap,id=net1,ifname=vun001000000c01,script=no -device vmxnet3,netdev=net2,mac=50:00:00:0c:00:02 -netdev tap,id=net2,ifname=vun001000000c02,script=no -device vmxnet3,netdev=net3,mac=50:00:00:0c:00:03 -netdev tap,id=net3,ifname=vun001000000c03,script=no -device vmxnet3,netdev=net4,mac=50:00:00:0c:00:04 -netdev tap,id=net4,ifname=vun001000000c04,script=no -device vmxnet3,netdev=net5,mac=50:00:00:0c:00:05 -netdev tap,id=net5,ifname=vun001000000c05,script=no -device vmxnet3,netdev=net6,mac=50:00:00:0c:00:06 -netdev tap,id=net6,ifname=vun001000000c06,script=no -device vmxnet3,netdev=net7,mac=50:00:00:0c:00:07 -netdev tap,id=net7,ifname=vun001000000c07,script=no -smp 2 -m 4096 -name RT-156 -uuid 3d9ab490-dbe0-4559-a3a0-7afd4b574e58 -qmp unix:./qmp-sock,server,nowait -monitor unix:./mon-sock,server,nowait -monitor unix:./mon2-sock,server,nowait -drive file=virtioa.qcow2,if=virtio,bus=0,unit=0,cache=none -machine type=pc,accel=kvm -cpu host,vmx=off,svm=off -serial mon:stdio -nographic -no-user-config -nodefaults -rtc base=utc
Running as unit: eveng_0@5db226d0-ea3f-4830-86aa-401614192b5d@12.service
Jul 19 15:06:39 INFO: starting systemctl show --property MainPID --value eveng_0@5db226d0-ea3f-4830-86aa-401614192b5d@12
Jul 19 15:06:39 INFO: started process is 2648018
PHP Warning: unlink(/opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/l1down_0): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1333
Jul 19 15:06:39 INFO: vun001000000c00 -> 0,0,0,0
PHP Warning: unlink(/opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/l1down_1): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1333
Jul 19 15:06:39 INFO: vun001000000c01 -> 0,0,0,0
PHP Warning: unlink(/opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/l1down_2): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1333
Jul 19 15:06:39 INFO: vun001000000c02 -> 0,0,0,0
PHP Warning: unlink(/opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/l1down_3): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1333
Jul 19 15:06:39 INFO: vun001000000c03 -> 0,0,0,0
PHP Warning: unlink(/opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/l1down_4): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1333
Jul 19 15:06:39 INFO: vun001000000c04 -> 0,0,0,0
PHP Warning: unlink(/opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/l1up_5): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1354
PHP Warning: unlink(/opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/suspend_5): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1356
PHP Warning: unlink(/opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/l1up_6): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1354
PHP Warning: unlink(/opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/suspend_6): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1356
PHP Warning: unlink(/opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/l1up_7): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1354
PHP Warning: unlink(/opt/unetlab/tmp/0/5db226d0-ea3f-4830-86aa-401614192b5d/12/suspend_7): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1356
Jul 19 15:06:39 Jul 19 15:06:39 ERROR: node 12