Community edition VM on ESXi 6.7 running on Dual Processor

[rusty725]

yes the last screenshot, is how I would set it. Are you on Eve Pro or Community ?

[mathewfer]

Hi Rusty,

Thanks. I am now trying the settings in the last screenshot and I am using EVE-ng Community Edition version 5.0.1-19.
I will update the progress in about 24hrs. So far for the last 45min or so, I did not see CPU errors or node freeze.

Thanks again for the very prompt reply - much appreciated.

[Uldis (UD)]

Best for you:
not sure if your server is supported by ESXi at all the best will be:
My server is:
Server PowerEdge R650
2x Intel(R) Xeon(R) Platinum 8368 CPU @ 2.40GHz (152 vCPU)
16x 32GB DDR4 DIMM Total 512
8x SAMSUNG SSD MZ7LH3T8HALT0D3 4TB
PERC H745

[mathewfer]

Hi UD/Rusty,

First thanks for the replies so far and they helped to improve.

There is a very significant improvement after setting suggested by Rusty. I do not see any CPU related error on EVE-ng logs now.
I have a heavy Linux VM running on EVE-ng and I increased CPUs to 12 and RAM to 16G in the Linux template when adding this Linux node and it gave few CPU errors. This same VM on EXSi on the same server runs with 4 CPUs and 8G ram without any CPU errors. This heavy Linux VM is used for DevOps running many network automations packages and I need to see it runs smoothly, without any freezing. Also I need to have a separate instance of this VM on each LAB. That is why I want to run it on EVE-ng.

I have pasted the errors from this Linux VM and if you have any ideas to improve it, please let me know.

I will monitor for this ESXi VM setting for next few days with the more and more LAB works and report here.

EVE-ng VM logs:

Code: Select all

root@eve-ng:~# cat /proc/cpuinfo | grep processor
processor       : 0
processor       : 1
processor       : 2
processor       : 3
processor       : 4
processor       : 5
processor       : 6
processor       : 7
processor       : 8
processor       : 9
processor       : 10
processor       : 11
processor       : 12
processor       : 13
processor       : 14
processor       : 15
processor       : 16
processor       : 17
processor       : 18
processor       : 19
processor       : 20
processor       : 21
processor       : 22
processor       : 23
processor       : 24
processor       : 25
processor       : 26
processor       : 27
processor       : 28
processor       : 29
processor       : 30
processor       : 31
processor       : 32
processor       : 33
processor       : 34
processor       : 35
processor       : 36
processor       : 37
processor       : 38
processor       : 39
processor       : 40
processor       : 41
processor       : 42
processor       : 43
processor       : 44
processor       : 45
processor       : 46
processor       : 47
processor       : 48
processor       : 49
processor       : 50
processor       : 51
processor       : 52
processor       : 53
processor       : 54
processor       : 55
processor       : 56
processor       : 57
processor       : 58
processor       : 59
processor       : 60
processor       : 61
processor       : 62
processor       : 63
processor       : 64
processor       : 65
processor       : 66
processor       : 67
processor       : 68
processor       : 69
root@eve-ng:~# cat /var/log/kern.log | grep watchdog
Dec  3 09:06:05 eve-ng kernel: [    1.063325] NMI watchdog: Perf NMI watchdog permanently disabled
Dec  5 09:47:40 eve-ng kernel: [    1.081549] NMI watchdog: Perf NMI watchdog permanently disabled
root@eve-ng:~#

DevOps Linux VM (running on EVE-ng) logs:

Code: Select all

user1@devops-srv:~$ cat /var/log/kern.log | grep watchdog
Dec  5 19:53:13 devops-srv kernel: NMI watchdog: Perf NMI watchdog permanently disabled
Dec  6 09:49:09 devops-srv kernel: watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [kworker/u24:4:47559]
Dec  6 09:49:09 devops-srv kernel:  ? watchdog_timer_fn+0x1be/0x220
Dec  6 09:49:37 devops-srv kernel: watchdog: BUG: soft lockup - CPU#4 stuck for 49s! [kworker/u24:4:47559]
Dec  6 09:49:37 devops-srv kernel:  ? watchdog_timer_fn+0x1be/0x220
Dec  6 09:50:05 devops-srv kernel: watchdog: BUG: soft lockup - CPU#4 stuck for 75s! [kworker/u24:4:47559]
Dec  6 09:50:05 devops-srv kernel:  ? watchdog_timer_fn+0x1be/0x220
user1@devops-srv:~$

[arjanbellyz]

I am very much confused when you say "Total overload of your CPU". I do not start all the nodes at the same time.
indigocard activate
roblox exploit

[mathewfer]

I do not start all the nodes at the same time. I do not go through the menu to start all.

I manually start one-by-one and that was my practice in the past few years with EVE-ng.

More -> I am still testing on ESXi, going mostly well so far, except few CPU errors/freezes, not on EVE-ng but on a Linux node and soon, I will use another 2T SSD to test bare-metal install EVE-ng CE and to run the full LAB to see the performance difference.

I will update here soon

[mathewfer]

After few days of testing by running the same LAB, I see CPU errors on EVE-ng. There were no change to LAB - no changes in any node configurations - LAB was just running.

Next, I will use new 2T SSD to install bare-metal to see how it goes and to isolate any ESXi related issues. If there is any hardware issues, it should produce the same errors.

EVE-ng:

Code: Select all

root@eve-ng:~# uptime
 09:54:30 up 3 days, 7 min,  1 user,  load average: 30.09, 28.57, 22.35
root@eve-ng:~#
root@eve-ng:~# cat /var/log/kern.log | grep watchdog
Dec  3 09:06:05 eve-ng kernel: [    1.063325] NMI watchdog: Perf NMI watchdog permanently disabled
Dec  5 09:47:40 eve-ng kernel: [    1.081549] NMI watchdog: Perf NMI watchdog permanently disabled
Dec  8 05:25:19 eve-ng kernel: [243476.747603] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [qemu-system-x86:134558]
Dec  8 05:36:23 eve-ng kernel: [244140.790766] watchdog: BUG: soft lockup - CPU#25 stuck for 26s! [qemu-system-x86:143548]
Dec  8 06:55:43 eve-ng kernel: [248900.849971] watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [migration/15:106]
Dec  8 06:55:47 eve-ng kernel: [248904.906053] watchdog: BUG: soft lockup - CPU#58 stuck for 26s! [qemu-system-x86:142666]
root@eve-ng:~#

Linux Node:

Code: Select all

Dec  8 02:23:38 tools-srv kernel: watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [processor:2320]
Dec  8 02:24:57 tools-srv kernel:  ? watchdog_timer_fn+0x1be/0x220
Dec  8 02:24:57 tools-srv kernel: watchdog: BUG: soft lockup - CPU#10 stuck for 21s! [processor:2320]
Dec  8 02:24:57 tools-srv kernel:  ? watchdog_timer_fn+0x1be/0x220
Dec  8 02:24:57 tools-srv kernel: watchdog: BUG: soft lockup - CPU#10 stuck for 47s! [processor:2320]
Dec  8 02:24:58 tools-srv kernel:  ? watchdog_timer_fn+0x1be/0x220
Dec  8 02:24:58 tools-srv kernel: watchdog: BUG: soft lockup - CPU#6 stuck for 102s! [ruby:1940]
Dec  8 02:24:58 tools-srv kernel:  ? watchdog_timer_fn+0x1be/0x220
Dec  8 05:13:50 tools-srv kernel: clocksource: timekeeping watchdog on CPU8: kvm-clock retried 2 times before success

[rusty725]

if you can get and install nvme, it may help.

[mathewfer]

Hi rusty

Thanks. I tried that by asking Supermicro how to get NVME as my board (https://www.supermicro.com/en/products/ ... d/X10DRL-i) does not support it natively.
This issues CPU issue actually started after I added the 2nd CPU and 64G RAM to that CPU to get Dual CPU with 128G RAM.

According to SuperMicro, if I add a PCI card with NVME slot, I can get max speed of 8G/s using x8x8 settings in the BIOS. Normal SSD supports 6G/s and with single CPU, I could run that Linux VM and small LABs, I did not have CPU errors.

I will do two tests:

1. Install bare metal and what is the stability
2. Install ESXi 6.7 on a new SSD as SSD in use is there for few years.

[saadnyce]

I am without answers and only way will be bare-metal install to test this. I can confirm that there is no hardware issues as all other VMs on this box runs without issues. I tested it after shutting down EVE-ng VM - I do not run all other VMs, except for one Linux VM for management of EVE-ng when running EVE-ng for LABs.
indigocard activate
jjsploit.click