satellite server not loading catalyst 9k vm

[kawegit]

I have a 3 server cluster.

When I try to run a Catalyst 9k node on a satellite server it fails. The image flashes for a minute, the "stopped" icon changes to a "play" icon for a sec like it's going to work, but a few seconds later I see the "stopped" icon again and the node never actually starts. The terminal never successfully connects either so I don't see any signs that the image actually boots.

I can however run Catalyst 9k vm's on the main server. I can also run lightweight iol images on the satellite server. It's only the heavy Catalyst 9k vm's running on satellite servers that fails.

Here are some logs of what I think is the main server trying to rsync the qemu image to the satellite server:


Apr 18 17:02:10 Apr 18 17:02:10 ERROR: node 1

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_24): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_24): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_23): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_23): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_22): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_22): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_21): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_21): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_20): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_20): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_19): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_19): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_18): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_18): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_17): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_17): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_16): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_16): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_15): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_15): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_14): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_14): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_13): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_13): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_12): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_12): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_11): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_11): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_10): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_10): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_9): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_9): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_8): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_8): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_7): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_7): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_6): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_6): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_5): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_5): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_4): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_4): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_3): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_3): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_2): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_2): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_1): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_1): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/suspend_0): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1332

PHP Warning: unlink(/opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1/l1up_0): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1330

}

string(2) "ok"

["status"]=>

string(7) "2823420"

["pid"]=>

int(200)

["code"]=>

Apr 18 17:02:10 INFO: array(3) {

Apr 18 17:02:08 INFO: starting /opt/unetlab/wrappers/qemu_wrapper -C 55607 -T 0 -D 1 -t "C9K" -F /opt/qemu-5.2.0/bin/qemu-system-x86_64 -d 0 -- -nographic -device e1000,netdev=net0,mac=50:00:00:01:00:00 -netdev tap,id=net0,ifname=vun001000000100,script=no -device e1000,netdev=net1,mac=50:00:00:01:00:01 -netdev tap,id=net1,ifname=vun001000000101,script=no -device e1000,netdev=net2,mac=50:00:00:01:00:02 -netdev tap,id=net2,ifname=vun001000000102,script=no -device e1000,netdev=net3,mac=50:00:00:01:00:03 -netdev tap,id=net3,ifname=vun001000000103,script=no -device e1000,netdev=net4,mac=50:00:00:01:00:04 -netdev tap,id=net4,ifname=vun001000000104,script=no -device e1000,netdev=net5,mac=50:00:00:01:00:05 -netdev tap,id=net5,ifname=vun001000000105,script=no -device e1000,netdev=net6,mac=50:00:00:01:00:06 -netdev tap,id=net6,ifname=vun001000000106,script=no -device e1000,netdev=net7,mac=50:00:00:01:00:07 -netdev tap,id=net7,ifname=vun001000000107,script=no -device e1000,netdev=net8,mac=50:00:00:01:00:08 -netdev tap,id=net8,ifname=vun001000000108,script=no -device e1000,netdev=net9,mac=50:00:00:01:00:09 -netdev tap,id=net9,ifname=vun001000000109,script=no -device e1000,netdev=net10,mac=50:00:00:01:00:0a -netdev tap,id=net10,ifname=vun00100000010a,script=no -device e1000,netdev=net11,mac=50:00:00:01:00:0b -netdev tap,id=net11,ifname=vun00100000010b,script=no -device e1000,netdev=net12,mac=50:00:00:01:00:0c -netdev tap,id=net12,ifname=vun00100000010c,script=no -device e1000,netdev=net13,mac=50:00:00:01:00:0d -netdev tap,id=net13,ifname=vun00100000010d,script=no -device e1000,netdev=net14,mac=50:00:00:01:00:0e -netdev tap,id=net14,ifname=vun00100000010e,script=no -device e1000,netdev=net15,mac=50:00:00:01:00:0f -netdev tap,id=net15,ifname=vun00100000010f,script=no -device e1000,netdev=net16,mac=50:00:00:01:00:10 -netdev tap,id=net16,ifname=vun001000000110,script=no -device e1000,netdev=net17,mac=50:00:00:01:00:11 -netdev tap,id=net17,ifname=vun001000000111,script=no -device e1000,netdev=net18,mac=50:00:00:01:00:12 -netdev tap,id=net18,ifname=vun001000000112,script=no -device e1000,netdev=net19,mac=50:00:00:01:00:13 -netdev tap,id=net19,ifname=vun001000000113,script=no -device e1000,netdev=net20,mac=50:00:00:01:00:14 -netdev tap,id=net20,ifname=vun001000000114,script=no -device e1000,netdev=net21,mac=50:00:00:01:00:15 -netdev tap,id=net21,ifname=vun001000000115,script=no -device e1000,netdev=net22,mac=50:00:00:01:00:16 -netdev tap,id=net22,ifname=vun001000000116,script=no -device e1000,netdev=net23,mac=50:00:00:01:00:17 -netdev tap,id=net23,ifname=vun001000000117,script=no -device e1000,netdev=net24,mac=50:00:00:01:00:18 -netdev tap,id=net24,ifname=vun001000000118,script=no -smp 4 -m 12288 -name C9K -uuid cb14f36f-2fe7-4789-a41b-a111b45b2991 -qmp unix:./qmp-sock,server,nowait -monitor unix:./mon-sock,server,nowait -monitor unix:./mon2-sock,server,nowait -drive file=virtioa.qcow2,if=virtio,bus=0,unit=0,cache=none -machine type=pc,accel=kvm -vga std -usbdevice tablet -boot order=cd -cdrom config.iso

Apr 18 17:02:08 INFO: CWD is /opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1Apr 18 17:02:08 INFO: CWD is /opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1

Apr 18 17:02:08 ERROR: QEMU Arch is not set (80015).

177 extents written (0 MB)

Max brk space used 0

Path table size(bytes): 10

Total directory bytes: 114

Total rockridge attributes bytes: 0

Total translation table size: 0

Apr 18 17:02:08 DEBUG: run command /opt/unetlab/config_scripts/init_cat9kvq200.sh /opt/unetlab/tmp/0/e2e31098-50d4-4efa-b012-8075ed221003/1

Apr 18 17:02:08 FLUSHING CACHE

pgrep -f drop_cache || ( sync && echo 3 > /proc/sys/vm/drop_caches )

Apr 18 17:02:08 STOP RSYNC

0inputs+8outputs (0major+643minor)pagefaults 0swaps

69.54user 0.81system 1:10.86elapsed 99%CPU (0avgtext+0avgdata 6376maxresident)k

total size is 2,739,404,800 speedup is 1.19

sent 2,311,073,448 bytes received 39 bytes 32,322,706.11 bytes/sec

cat9kvq200-17.12.01-prd9/virtioa.qcow2

cat9kvq200-17.12.01-prd9/

sending incremental file list

time rsync -e 'ssh -o ControlPath=/tmp/ssh-root@172.29.130.1:22 -o StrictHostKeyChecking=no' -azv /opt/unetlab/addons/qemu/cat9kvq200-17.12.01-prd9 172.29.130.1:/opt/unetlab/addons/qemu/ >> /opt/unetlab/data/Logs/unl_wrapper.txt 2>&1

Apr 18 17:00:57 START RSYNC

[rusty725]

there's going to be an update sometime soon

[kawegit]

update fixed it, thank you