Heavy writes caused by process-scanner

Moderator: mike

Post Reply
janegil
Posts: 5
Joined: Sun Feb 10, 2019 7:50 pm

Heavy writes caused by process-scanner

Post by janegil » Thu Sep 01, 2022 6:12 pm

Hi

I have noticed high disk-utilization (many writes) by mysql process with no labs/nodes running. This causes the whole system to get less responsive.
These writes seems to come from "process-scanner" which causes high-frequent updates to the mysql disk_usage table. Stopping the process-scanner (or mysql) solves this problem.
Looking at performance with atop the disk-activity raises with more than 40% in some of my cases. ('atop' and 'atopsar -d 10') when this process is running.
This has been tested on virtual machines and bare-host installation with the eve-ng-pro on machines with NVMe SSDs.

By doing a mysql full query log I have found that there are a lot of updates to the disk_usage table (17760 updates in 3-4 minutes).

Code: Select all

$ cat query.log | grep "update disk_usage" | wc -l
   17760
   
$ cat query.log | grep "update disk_usage" | head -n 5
2022-09-01T07:05:14.546543Z	    8 Query	update disk_usage set checked = 0 where sat = 0
2022-09-01T07:05:14.633778Z	    8 Query	update disk_usage set size = 1173958656 , checked = 1 where podId = 1 and labUuid = '194174de-160f-4860-be7e-2dbed92589c1' and nodeId = 4 and sat = 0
2022-09-01T07:05:14.640368Z	    8 Query	update disk_usage set size = 1136275456 , checked = 1 where podId = 1 and labUuid = '194174de-160f-4860-be7e-2dbed92589c1' and nodeId = 3 and sat = 0
2022-09-01T07:05:14.647715Z	    8 Query	update disk_usage set size = 1130442752 , checked = 1 where podId = 1 and labUuid = '194174de-160f-4860-be7e-2dbed92589c1' and nodeId = 2 and sat = 0
2022-09-01T07:05:14.655892Z	    8 Query	update disk_usage set size = 1113468928 , checked = 1 where podId = 1 and labUuid = '194174de-160f-4860-be7e-2dbed92589c1' and nodeId = 1 and sat = 0

$ cat query.log | grep "update disk_usage" | tail -n 5
2022-09-01T07:08:58.655307Z	    8 Query	update disk_usage set size = 1064960 , checked = 1 where podId = 1 and labUuid = '01845860-4a17-419b-9cd4-c4b782eb874f' and nodeId = 1 and sat = 0
2022-09-01T07:08:58.661700Z	    8 Query	update disk_usage set size = 270336 , checked = 1 where podId = 1 and labUuid = '42e369f2-9ff6-49b5-9f1f-a0863d89a513' and nodeId = 1 and sat = 0
2022-09-01T07:08:58.667941Z	    8 Query	update disk_usage set size = 208896 , checked = 1 where podId = 1 and labUuid = '3f2b9655-38b4-405e-aa40-f790b344349e' and nodeId = 1 and sat = 0
2022-09-01T07:08:58.674465Z	    8 Query	update disk_usage set size = 16384 , checked = 1 where podId = 1 and labUuid = '3f2b9655-38b4-405e-aa40-f790b344349e' and nodeId = 2 and sat = 0
2022-09-01T07:08:58.683009Z	    8 Query	update disk_usage set size = 16384 , checked = 1 where podId = 1 and labUuid = '9f6752fa-4c62-4bb6-b34d-3cf8ce16eca8' and nodeId = 2 and sat = 0
'inotifywait -e modify -e attrib -e move -e create -e delete -m -r /var/lib/mysql' shows alot of activity on the files for log-bin and redo log. Disabling these (binlog and redo log) in mysql as a test gets rid of the problem:

Code: Select all

mysql> ALTER INSTANCE DISABLE INNODB REDO_LOG;
/etc/mysql/mysql.conf.d/mysqld.cnf: disable-log-bin
Moving files away from /opt/unetlab/tmp also decreases write-activity (as there are less entries to update). I can also generate many empty qcow files in the tmp directory and make disk-activity permanently increase as a result

Code: Select all

# Decrease writes
mv /opt/unetlab/tmp/1 /opt/unetlab/tmp/temp) 

# Increase writes (has more effect with bigger files as normally exists)
cd /opt/unetlab/tmp ; for i in {1..200}; do uuid=$(uuid); mkdir -p "10/$uuid/1"; touch "10/$uuid/1/virtioa.qcow2"; done
The amount of slowness this problems generates, and the possible reduction of lifetime on SSDs can't in my opinion be worth the benefit of having second-fresh disk-usage numbers in the database (which is probably only used in the GUI?)
Maybe this can be optimized in a way? Store this information somewhere else, reduce the update frequency or update mysql in larger transactions?

I have also earlier reduced the log-bin retention as my logbin was huge. I suspect this also to be an impact of the disk_usage updates as the rest of the information seems to be of much more static nature.

Code: Select all

/etc/mysql/mysql.conf.d/mysqld.cnf:binlog_expire_logs_seconds      = 172800
(I have tried doing a fresh install on a different host, rehosted the license and moved my files according to the documents, which did not seem to make a difference.)

Edit:
With versions

Code: Select all

root@eve-ng:~# uname -a; dpkg -l | grep eve-ng-pro;
Linux eve-ng 5.17.8-eve-ng-uksm-wg+ #1 SMP PREEMPT Mon May 16 10:08:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
ii  eve-ng-pro                            5.0.1-20                              amd64        A new generation software for networking labs.

Post Reply