We're having an issue with our eve-ng platform, where capture does not work anymore (it has worked smoothly for months until today).
No change has been made as fas I as know.
(See full eve-info at the end of this post)
When attempting to capture on a link, we get a pop-up saying "you have been disconnected" :
In unl_wrappers logs I see the following errors at the same time:
Code: Select all
Error: No such object: Capture-8390659
Error: Parent Qdisc doesn't exists.
We have an error talking to the kernel
Error: Parent Qdisc doesn't exists.
We have an error talking to the kernel
Error: Invalid handle.
Feb 05 11:06:35 INFO: starting docker -H=tcp://127.0.0.1:4243 create --shm-size 1G --env GDK_SCALE=2 --env QT_SCALE_FACTOR=1.5 --device /dev/fuse --cap-add=SYS_ADMIN --cap-add=NET_ADMIN -ti --net=none --name=Capture-8390659 -h Capture-vun008000000203 eve-wireshark
Feb 05 11:06:35 INFO: starting docker -H=tcp://127.0.0.1:4243 start Capture-8390659
Feb 05 11:06:36 INFO: started process is
Feb 05 11:06:36 INFO: starting ip link delete cap_8390659
Cannot find device "cap_8390659"
Feb 05 11:06:36 INFO: starting ip link add dcap_8390659 type veth peer name cap_8390659
Feb 05 11:06:36 INFO: starting ip link set dev cap_8390659 up
Feb 05 11:06:36 INFO: sysctl -w net.ipv6.conf.cap_8390659.disable_ipv6=1
Feb 05 11:06:36 INFO: sysctl -w net.ipv6.conf.dcap_8390659.disable_ipv6=1
Feb 05 11:06:36 INFO: starting docker -H=tcp://127.0.0.1:4243 inspect --format "{{ .State.Pid }}" Capture-8390659
Feb 05 11:06:36 INFO: starting ip link delete rdp_8390659
Cannot find device "rdp_8390659"
Feb 05 11:06:36 INFO: starting brctl addif docker0 rdp_8390659
Feb 05 11:06:36 INFO: starting docker -H=tcp://127.0.0.1:4243 inspect --format "{{ .State.Pid }}" Capture-8390659
Feb 05 11:06:36 INFO: starting ip link set netns 3814083 dock0_8390659 name eth1 address 50:00:00:80:08:03 up
Feb 05 11:06:36 INFO: starting /opt/unetlab/wrappers/nsenter -t 3814083 -n ip addr add 172.17.178.139/16 dev eth1
Feb 05 11:06:36 INFO: starting /opt/unetlab/wrappers/nsenter -t 3814083 -n sysctl -w net.ipv6.conf.eth1.disable_ipv6=1
mkdir: cannot create directory '/tmp/admin': File exists
chown: invalid user: 'admin'
Code: Select all
root@nieeve03:/opt/unetlab/data/Logs# dc images
REPOSITORY TAG IMAGE ID CREATED SIZE
eve-gui-server latest d819486ab729 10 months ago 7.07GB
eve-desktop latest aaab99abf9b9 10 months ago 6.88GB
eve-firefox latest 01c3151ae759 11 months ago 4.69GB
eve-wireshark latest 030d66992f3d 11 months ago 4.24GB
registry 2 cfb4d9904335 16 months ago 25.4MB
alpine latest b2aa39c304c2 24 months ago 7.05MB
registry <none> 773dbf02e42e 2 years ago 24.1MB
wbitt/network-multitool latest 0e29af756369 3 years ago 67.9MB
cturra/ntp latest 3ae140972557 3 years ago 6.07MB
bgp-full-routing latest 40a84eefa9bd 4 years ago 11GB
ubuntu 18.04 c090eaba6b94 4 years ago 63.3MB
eve-ostinato latest d8a6585a73c4 4 years ago 1.61GB
eve-kali-large latest 3b161f88d472 5 years ago 12GB
ntpd/ntpd latest bca676b07b82 8 years ago 14.1MB
freva/simple-ipv6-server latest 94d1eabb92e9 8 years ago 44.8MB
I thought about reinstalling eve-ng-dockers but I am wondering wether this will cause any disruption?
I am not sure what else to do..
This is a heavily used cluster with many nodes running so if possible, I would like to avoid doing anything disruptive.
Many thanks in advance for your help with this issue.
Best,
Alex
eve-info:
Code: Select all
root@nieeve03:/opt/unetlab/data/Logs# eve-info
Wed Feb 5 11:12:25 AM CET 2025
---------------Packages Installed----------------
ii eve-ng-addons-ostinato-drone 5.0.1-21
ii eve-ng-dockers 6.0-eve-ng
ii eve-ng-dynamips 6.0.1-5
ii eve-ng-pro 6.2.0-6
ii eve-ng-pro-guacamole 6.0.1-37
ii eve-ng-qemu 6.0.1-0
ii eve-ng-schema 6.0.1-0
ii eve-ng-vpcs 6.1-eve-ng
ii linux-image-4.20.17-eve-ng-uksm+ 1
ii linux-image-4.20.17-eve-ng-uksm-wg+ 1
ii linux-image-4.20.17.2-eve-ng-uksm-wg+ 1
ii linux-image-5.17.8-eve-ng-uksm-wg+ 1
---------------Hostname--------------------------
Static hostname: nieeve03
Operating System: Ubuntu 22.04.4 LTS
Kernel: Linux 6.7.5-eveng-6-ksm+
Architecture: x86-64
---------------Disk Usage------------------------
Filesystem Size Used Avail Use% Mounted on
tmpfs 32G 11M 32G 1% /run
/dev/mapper/nieeve03--vg-root 1.7T 729G 936G 44% /
tmpfs 158G 588K 158G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 158G 0 158G 0% /run/qemu
tmpfs 32G 0 32G 0% /run/user/0
---------------CPU Info--------------------------
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 56
On-line CPU(s) list: 0-55
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz
CPU family: 6
Model: 63
Thread(s) per core: 2
Core(s) per socket: 14
Socket(s): 2
Stepping: 2
CPU max MHz: 3600.0000
CPU min MHz: 1200.0000
BogoMIPS: 5199.48
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nop
l xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave a
vx f16c rdrand lahf_lm abm cpuid_fault epb ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dthe
rm ida arat pln pts vnmi md_clear flush_l1d
Virtualization features:
Virtualization: VT-x
Caches (sum of all):
L1d: 896 KiB (28 instances)
L1i: 896 KiB (28 instances)
L2: 7 MiB (28 instances)
L3: 70 MiB (2 instances)
NUMA:
NUMA node(s): 2
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55
Vulnerabilities:
Gather data sampling: Not affected
Itlb multihit: KVM: Vulnerable
L1tf: Mitigation; PTE Inversion; VMX vulnerable
Mds: Vulnerable; SMT vulnerable
Meltdown: Vulnerable
Mmio stale data: Vulnerable
Retbleed: Not affected
Spec rstack overflow: Not affected
Spec store bypass: Vulnerable
Spectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
Spectre v2: Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBRS: Not affected
Srbds: Not affected
Tsx async abort: Not affected
---------------Memory Info-----------------------
total used free shared buff/cache available
Mem: 314Gi 97Gi 80Gi 80Mi 136Gi 214Gi
Swap: 975Mi 617Mi 358Mi
---------------Nic Info--------------------------
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master pnet0 state UP mode DEFAULT group default qlen 1000
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master pnet1 state UP mode DEFAULT group default qlen 1000
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq master pnet2 state UP mode DEFAULT group default qlen 1000
5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq master pnet3 state UP mode DEFAULT group default qlen 1000
6: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq master pnet4 state UP mode DEFAULT group default qlen 1000
7: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq master pnet5 state UP mode DEFAULT group default qlen 1000
8: eth6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq master pnet6 state UP mode DEFAULT group default qlen 1000
9: eth7: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master pnet7 state DOWN mode DEFAULT group default qlen 1000
---------------IP Info---------------------------
● State: n/a
Online state: unknown
172.29.129.254 on nat0
172.17.0.1 on docker0
172.29.130.254 on wg0
---------------Bridge Info-----------------------
pnet0 8000.246e96097054 no eth0
pnet1 8000.5a490288b830 no eth1
pnet2 8000.62a163662089 no eth2
pnet3 8000.3a4db57733e4 no eth3
pnet4 8000.d6090961074c no eth4
pnet5 8000.26afc1b1057f no eth5
pnet6 8000.3af7876c3527 no eth6
pnet7 8000.d66f05445f46 no eth7
pnet8 8000.fefb257b8435 no
pnet9 8000.0eef28f819c2 no
---------------H/W Accel-------------------------
INFO: /dev/kvm exists
KVM acceleration can be used
---------------Service Info----------------------
-------------------------------------------------
--------------Guacamole--------------------------
● guacd.service - LSB: Guacamole proxy daemon
Loaded: loaded (/etc/init.d/guacd; generated)
Active: active (running) since Wed 2024-08-21 18:19:06 CEST; 5 months 15 days ago
--------------Tomcat-----------------------------
Unit tomcat8.service could not be found.
--------------Mysql------------------------------
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2024-08-21 18:19:18 CEST; 5 months 15 days ago
--------------Apache-----------------------------
● apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/apache2.service.d
└─local.conf
Active: active (running) since Wed 2024-08-21 18:19:08 CEST; 5 months 15 days ago