Nodes stopped starting

Before posting something, READ the changelog, WATCH the videos, howto and provide following:
Your install is: Bare metal, ESXi, what CPU model, RAM, HD, what EVE version you have, output of the uname -a and any other info that might help us faster.

Moderator: mike

Locked
maxxus
Posts: 11
Joined: Wed May 17, 2023 6:35 pm

Nodes stopped starting

Post by maxxus » Wed May 17, 2023 7:05 pm

Hi!
I have a problem: nodes do not start in eve-ng.
Description:
I used Free EVE Community Edition Version 5.0.1-19
set up a multi-nodes scheme in EVE-NG.
They turned on and worked.
Then I turned off all nodes.
Removed the Linux node.
Deleted his image /opt/unetlab/addons/qemu/linux-debian11/virtioa.qcow2
Uploaded a new image (with additional software)
Ran /opt/unetlab/wrappers/unl_wrapper -a fixpermissions
Added a Linux node to the lab and...
from that moment on, all nodes stopped starting.
In Notifications I see the Started node, but nothing happens.
There are no errors in the logs, in unl_wrapper.txt I see that the nodes are starting up and nothing else happens.

Code: Select all

tail -f  unl_wrapper.txt

May 17 21:02:18 INFO: tunctl -u unl0 -g root -t vunl0_1_0 2>&1
May 17 21:02:18 INFO: sysctl -w net.ipv6.conf.vunl0_1_0.disable_ipv6=1
May 17 21:02:18 cmd = mountpoint -q /opt/unetlab/tmp/0/be323e0f-e06e-4553-bd39-67234005aa76/1/jail || mount -B -o ro /opt/unetlab/jail /opt/unetlab/tmp/0/be323e0f-e06e-4553-bd39-67234005aa76/1/jail
May 17 21:02:18 cmd = for i in /opt/qemu-*; do ( mountpoint -q /opt/unetlab/tmp/0/be323e0f-e06e-4553-bd39-67234005aa76/1/$i || mount -B -o ro $i /opt/unetlab/tmp/0/be323e0f-e06e-4553-bd39-67234005aa76/1/$i ) ; done 
May 17 21:02:18 cmd =  mountpoint -q /opt/unetlab/tmp/0/be323e0f-e06e-4553-bd39-67234005aa76/1/opt/unetlab/addons/qemu || mount -B -o ro /opt/unetlab/addons/qemu /opt/unetlab/tmp/0/be323e0f-e06e-4553-bd39-67234005aa76/1/opt/unetlab/addons/qemu
May 17 21:02:18 INFO: CWD is /opt/unetlab/tmp/0/be323e0f-e06e-4553-bd39-67234005aa76/1
May 17 21:02:18 INFO: starting /opt/unetlab/wrappers/qemu_wrapper -T 0 -D 1 -t "Linux" -F /opt/qemu-2.12.0/bin/qemu-system-x86_64 -d 0 -x -- -device virtio-net-pci,netdev=net0,mac=00:50:00:00:01:00 -netdev tap,id=net0,ifname=vunl0_1_0,script=no -smp 2 -m 4096 -name Linux -uuid 8058a404-4afc-4ac4-94f3-76df84bbf631 -vnc :26869 -drive file=virtioa.qcow2,if=virtio,bus=0,unit=0,cache=none -machine type=pc,accel=kvm -vga std -usbdevice tablet -boot order=cd -cpu host > /opt/unetlab/tmp/0/be323e0f-e06e-4553-bd39-67234005aa76/1/wrapper.txt 2>&1 &
Please help me with this problem
P.S: Sorry for my english, it's not my native language

maxxus
Posts: 11
Joined: Wed May 17, 2023 6:35 pm

Re: Nodes stopped starting

Post by maxxus » Tue Jun 06, 2023 12:33 pm

After reinstalling eve-ng, the problem did not appear until now...
For no apparent reason, the nodes stopped running...
in addition to the main post, I found in the logs:

Code: Select all

cat /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/7/wrapper.txt
6/5 12:4:11.875 INF     Tennant_id = 0
6/5 12:4:11.875 INF     Device_id = 7
6/5 12:4:11.878 ERR     Child is no more running.

maxxus
Posts: 11
Joined: Wed May 17, 2023 6:35 pm

Re: Nodes stopped starting

Post by maxxus » Wed Jun 07, 2023 2:28 pm

I'm still looking for a problem.
I saw that when starting any node, the disk partition becomes read only:

Code: Select all

/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/jail type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/qemu-1.3.1 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/qemu-2.0.2 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/qemu-2.12.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/qemu-2.2.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/qemu-2.4.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/qemu-2.5.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/qemu-2.6.2 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/qemu-3.1.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/qemu-4.1.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/qemu-5.2.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/qemu-6.0.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/unetlab/addons/qemu type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/jail type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/qemu-1.3.1 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/qemu-2.0.2 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/qemu-2.12.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/qemu-2.2.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/qemu-2.4.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/qemu-2.5.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/qemu-2.6.2 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/qemu-3.1.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/qemu-4.1.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/qemu-5.2.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/qemu-6.0.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/unetlab/addons/qemu type ext4 (ro,relatime

If anyone has any thoughts on this, please share.

maxxus
Posts: 11
Joined: Wed May 17, 2023 6:35 pm

Re: Nodes stopped starting

Post by maxxus » Wed Jun 07, 2023 3:03 pm

maxxus wrote:
Wed Jun 07, 2023 2:28 pm
I'm still looking for a problem.
I saw that when starting any node, the disk partition becomes read only:

Code: Select all

/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/jail type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/qemu-1.3.1 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/qemu-2.0.2 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/qemu-2.12.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/qemu-2.2.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/qemu-2.4.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/qemu-2.5.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/qemu-2.6.2 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/qemu-3.1.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/qemu-4.1.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/qemu-5.2.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/qemu-6.0.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/13/opt/unetlab/addons/qemu type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/jail type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/qemu-1.3.1 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/qemu-2.0.2 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/qemu-2.12.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/qemu-2.2.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/qemu-2.4.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/qemu-2.5.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/qemu-2.6.2 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/qemu-3.1.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/qemu-4.1.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/qemu-5.2.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/qemu-6.0.0 type ext4 (ro,relatime)
/dev/mapper/VG_OPT-opt on /opt/unetlab/tmp/0/069665fb-f07a-47c8-8031-166c4ce97400/2/opt/unetlab/addons/qemu type ext4 (ro,relatime

If anyone has any thoughts on this, please share.

Update:
this is the result of regular work.
keep searching

maxxus
Posts: 11
Joined: Wed May 17, 2023 6:35 pm

Re: Nodes stopped starting

Post by maxxus » Fri Jun 09, 2023 8:16 am

The only thing I found in the logs:

Code: Select all

cat /opt/unetlab/data/Logs/php_errors.txt 
[09-Jun-2023 10:10:06 Europe/Rome] Jun 09 10:10:06 WARNING: Attribute ignored, invalid node_uuid (40026).
there are no more errors

maxxus
Posts: 11
Joined: Wed May 17, 2023 6:35 pm

Re: Nodes stopped starting

Post by maxxus » Tue Jun 13, 2023 1:50 pm

after increasing the delay on the node to 30,
I saw error in node's console:

Code: Select all

chroot: failed to run command '/opt/qemu-4.1.0/bin/qemu-system-x86_64': Permission denied
13/5 12:Connection closed by foreign host.
I still don't understand why this problem occurs.

maxxus
Posts: 11
Joined: Wed May 17, 2023 6:35 pm

Re: Nodes stopped starting

Post by maxxus » Fri Jun 23, 2023 1:15 pm

Problem solved.
The problem occurred after running /opt/unetlab/wrappers/unl_wrapper -a fixpermission.
It was necessary to additionally change the permissions for files in the /opt/unetlab/jail/ directory:
chmod -R 755 /opt/unetlab/jail
this command is added to the script
/opt/unetlab/wrappers/unl_wrapper -a fixpermissions
and now everything works

P.S. Although, I don’t understand why no one except me noticed such a problem

Locked