I have noticed high disk-utilization (many writes) by mysql process with no labs/nodes running. This causes the whole system to get less responsive.
These writes seems to come from "process-scanner" which causes high-frequent updates to the mysql disk_usage table. Stopping the process-scanner (or mysql) solves this problem.
Looking at performance with atop the disk-activity raises with more than 40% in some of my cases. ('atop' and 'atopsar -d 10') when this process is running.
This has been tested on virtual machines and bare-host installation with the eve-ng-pro on machines with NVMe SSDs.
By doing a mysql full query log I have found that there are a lot of updates to the disk_usage table (17760 updates in 3-4 minutes).
Code: Select all
$ cat query.log | grep "update disk_usage" | wc -l
17760
$ cat query.log | grep "update disk_usage" | head -n 5
2022-09-01T07:05:14.546543Z 8 Query update disk_usage set checked = 0 where sat = 0
2022-09-01T07:05:14.633778Z 8 Query update disk_usage set size = 1173958656 , checked = 1 where podId = 1 and labUuid = '194174de-160f-4860-be7e-2dbed92589c1' and nodeId = 4 and sat = 0
2022-09-01T07:05:14.640368Z 8 Query update disk_usage set size = 1136275456 , checked = 1 where podId = 1 and labUuid = '194174de-160f-4860-be7e-2dbed92589c1' and nodeId = 3 and sat = 0
2022-09-01T07:05:14.647715Z 8 Query update disk_usage set size = 1130442752 , checked = 1 where podId = 1 and labUuid = '194174de-160f-4860-be7e-2dbed92589c1' and nodeId = 2 and sat = 0
2022-09-01T07:05:14.655892Z 8 Query update disk_usage set size = 1113468928 , checked = 1 where podId = 1 and labUuid = '194174de-160f-4860-be7e-2dbed92589c1' and nodeId = 1 and sat = 0
$ cat query.log | grep "update disk_usage" | tail -n 5
2022-09-01T07:08:58.655307Z 8 Query update disk_usage set size = 1064960 , checked = 1 where podId = 1 and labUuid = '01845860-4a17-419b-9cd4-c4b782eb874f' and nodeId = 1 and sat = 0
2022-09-01T07:08:58.661700Z 8 Query update disk_usage set size = 270336 , checked = 1 where podId = 1 and labUuid = '42e369f2-9ff6-49b5-9f1f-a0863d89a513' and nodeId = 1 and sat = 0
2022-09-01T07:08:58.667941Z 8 Query update disk_usage set size = 208896 , checked = 1 where podId = 1 and labUuid = '3f2b9655-38b4-405e-aa40-f790b344349e' and nodeId = 1 and sat = 0
2022-09-01T07:08:58.674465Z 8 Query update disk_usage set size = 16384 , checked = 1 where podId = 1 and labUuid = '3f2b9655-38b4-405e-aa40-f790b344349e' and nodeId = 2 and sat = 0
2022-09-01T07:08:58.683009Z 8 Query update disk_usage set size = 16384 , checked = 1 where podId = 1 and labUuid = '9f6752fa-4c62-4bb6-b34d-3cf8ce16eca8' and nodeId = 2 and sat = 0
Code: Select all
mysql> ALTER INSTANCE DISABLE INNODB REDO_LOG;
/etc/mysql/mysql.conf.d/mysqld.cnf: disable-log-bin
Code: Select all
# Decrease writes
mv /opt/unetlab/tmp/1 /opt/unetlab/tmp/temp)
# Increase writes (has more effect with bigger files as normally exists)
cd /opt/unetlab/tmp ; for i in {1..200}; do uuid=$(uuid); mkdir -p "10/$uuid/1"; touch "10/$uuid/1/virtioa.qcow2"; done
Maybe this can be optimized in a way? Store this information somewhere else, reduce the update frequency or update mysql in larger transactions?
I have also earlier reduced the log-bin retention as my logbin was huge. I suspect this also to be an impact of the disk_usage updates as the rest of the information seems to be of much more static nature.
Code: Select all
/etc/mysql/mysql.conf.d/mysqld.cnf:binlog_expire_logs_seconds = 172800
Edit:
With versions
Code: Select all
root@eve-ng:~# uname -a; dpkg -l | grep eve-ng-pro;
Linux eve-ng 5.17.8-eve-ng-uksm-wg+ #1 SMP PREEMPT Mon May 16 10:08:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
ii eve-ng-pro 5.0.1-20 amd64 A new generation software for networking labs.