Kernel Panic when running 2 Vendors at the same time

Before posting something, READ the changelog, WATCH the videos, howto and provide following:
Your install is: Bare metal, ESXi, what CPU model, RAM, HD, what EVE version you have, output of the uname -a and any other info that might help us faster.

Moderator: mike

Post Reply
eboman
Posts: 3
Joined: Fri Mar 20, 2020 12:00 pm

Kernel Panic when running 2 Vendors at the same time

Post by eboman » Fri Mar 20, 2020 12:13 pm

Hardware : Azure VM (Standard E20s_v3 (20 vcpus, 160 GiB memory) - i know this is not supported because of the kernel
EVE-NG Running on - Linux (ubuntu 16.04)

I have an issue where i have an Cisco IOL L3 running with an VSR1000 at the same time (even in a lab with one of each), i will get an kernel panic.

Both can run separate in a bigger lab without issues , but starting 1 router of the other vendor the panic will start.

Images i use :
hpvsr-710-CMV710-R0327L01
l3-adventerprisek9-15.4.2T.bin

Eve-ng community version : 2.0.3.-105
Qemu version 2.4.0

Maybe someone can try to replicate ? In the meanwhile i will try other versions of the images.

eboman
Posts: 3
Joined: Fri Mar 20, 2020 12:00 pm

Re: Kernel Panic when running 2 Vendors at the same time

Post by eboman » Fri Mar 20, 2020 1:39 pm

I got some info from the console :
[ 3358.605398] PANIC: double fault, error_code: 0x0
[ 3358.608607] Kernel panic - not syncing: Machine halted.
[ 3358.608806] CPU: 15 PID: 41341 Comm: l2-adventerpris Not tainted 4.20.17-eve-ng-ukms+ #2
[ 3358.608806] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007 06/02/2017
[ 3358.608806] Call Trace:
[ 3358.608806] <#DF>
[ 3358.608806] dump_stack+0x63/0x85
[ 3358.608806] panic+0xfe/0x264
[ 3358.608806] df_debug+0x2d/0x30
[ 3358.608806] do_double_fault+0x9a/0x130
[ 3358.608806] double_fault+0x1e/0x30
[ 3358.608806] RIP: 0010:0x1520
[ 3358.608806] Code: Bad RIP value.
[ 3358.608806] RSP: 0018:0000000000007200 EFLAGS: 00010096
[ 3358.608806] RAX: 0000000000000102 RBX: 00000000f7b499e8 RCX: 00000000ffb06f74
[ 3358.608806] RDX: 000000000000002f RSI: 00000000f7b49980 RDI: 00000000f7f5c000
[ 3358.608806] RBP: 00000000ffb06bf0 R08: 0000000000000000 R09: 0000000000000000
[ 3358.608806] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 3358.664188] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 3358.664188] </#DF>
[ 3358.664188] Kernel Offset: 0x33200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 3358.664188] ---[ end Kernel panic - not syncing: Machine halted. ]---
[ 3358.674766] ------------[ cut here ]------------
[ 3358.674766] sched: Unexpected reschedule of offline CPU#9!
[ 3358.674766] WARNING: CPU: 15 PID: 41341 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x3f/0x50
[ 3358.674766] Modules linked in: openvswitch nsh nf_nat_ipv6 nf_nat_ipv4 nf_conncount nf_nat xt_owner xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_security ip_tables x_tables bpfilter nls_iso8859_1 mlx4_en mlx4_core devlink bridge sb_edac stp llc i2c_piix4 pci_hyperv kvm_intel input_leds kvm joydev hv_balloon serio_raw mac_hid irqbypass intel_rapl_perf ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul hid_generic crc32_pclmul hid_hyperv hyperv_fb hv_storvsc cfbfillrect ghash_clmulni_intel cfbimgblt hyperv_keyboard hid cfbcopyarea hv_netvsc scsi_transport_fc hv_utils aesni_intel aes_x86_64 crypto_simd ide_pci_generic cryptd piix glue_helper psmouse fb ide_core i2c_core fbdev pata_acpi hv_vmbus floppy
[ 3358.674766] CPU: 15 PID: 41341 Comm: l2-adventerpris Not tainted 4.20.17-eve-ng-ukms+ #2
[ 3358.674766] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007 06/02/2017
[ 3358.674766] RIP: 0010:native_smp_send_reschedule+0x3f/0x50
[ 3358.674766] Code: c0 84 c0 74 17 48 8b 05 0f f5 13 01 be fd 00 00 00 48 8b 40 30 e8 f1 2a ba 00 5d c3 89 fe 48 c7 c7 58 f5 2a b5 e8 41 65 03 00 <0f> 0b 5d c3 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00
[ 3358.674766] RSP: 0018:ffff8a8efdfc3c58 EFLAGS: 00010086
[ 3358.674766] RAX: 0000000000000000 RBX: 0000000000000009 RCX: 0000000000000006
[ 3358.674766] RDX: 0000000000000007 RSI: 0000000000000086 RDI: ffff8a8efdfd6440
[ 3358.674766] RBP: ffff8a8efdfc3c58 R08: 0000000000000001 R09: 0000000000000000
[ 3358.674766] R10: 00000000000a11ef R11: 0000000000000038 R12: ffff8a8ea513ae00
[ 3358.674766] R13: ffff8a8efde62cc0 R14: ffff8a8efdfc3d10 R15: ffff8a8efde62cc0
[ 3358.674766] FS: 0000000000000000(0000) GS:ffff8a8efdfc0000(0063) knlGS:00000000f7b49980
[ 3358.674766] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033
[ 3358.674766] CR2: 00000000000014f6 CR3: 00000027cd2c8001 CR4: 00000000003626e0
[ 3358.674766] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 3358.674766] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 3358.674766] Call Trace:
[ 3358.674766] <IRQ>
[ 3358.674766] resched_curr+0x6c/0xd0
[ 3358.674766] check_preempt_curr+0x54/0x90
[ 3358.674766] ttwu_do_wakeup+0x1e/0x150
[ 3358.674766] ttwu_do_activate+0x77/0x80
[ 3358.674766] try_to_wake_up+0x1db/0x530
[ 3358.674766] default_wake_function+0x12/0x20
[ 3358.674766] autoremove_wake_function+0x12/0x40
[ 3358.674766] __wake_up_common+0x8c/0x130
[ 3358.674766] __wake_up_common_lock+0x80/0xc0
[ 3358.674766] __wake_up+0x13/0x20
[ 3358.674766] wake_up_klogd_work_func+0x40/0x60
[ 3358.674766] irq_work_run_list+0x55/0x80
[ 3358.674766] ? tick_sched_do_timer+0x60/0x60
[ 3358.674766] irq_work_tick+0x40/0x50
[ 3358.674766] update_process_times+0x42/0x60
[ 3358.674766] tick_sched_handle+0x29/0x60
[ 3358.674766] tick_sched_timer+0x3c/0x80
[ 3358.674766] __hrtimer_run_queues+0x106/0x270
[ 3358.674766] hrtimer_interrupt+0x116/0x240
[ 3358.674766] hv_stimer0_isr+0x24/0x40 [hv_vmbus]
[ 3358.674766] hv_stimer0_vector_handler+0x3f/0x70
[ 3358.674766] hv_stimer0_callback_vector+0xf/0x20
[ 3358.674766] </IRQ>
[ 3358.674766] <#DF>
[ 3358.674766] RIP: 0010:panic+0x21b/0x264
[ 3358.674766] Code: eb a6 83 3d 9a ec 88 01 00 74 05 e8 73 77 02 00 48 c7 c6 e0 41 b2 b5 48 c7 c7 88 92 2b b5 e8 83 8e 06 00 fb 66 0f 1f 44 00 00 <31> db e8 a2 d9 0d 00 4c 39 eb 7c 1d 41 83 f4 01 48 8b 05 42 ec 88
[ 3358.674766] RSP: 0018:fffffe000028be90 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff12
[ 3358.674766] RAX: 0000000000000039 RBX: fffffe000028bf00 RCX: 0000000000000006
[ 3358.674766] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff8a8efdfd6440
[ 3358.674766] RBP: fffffe000028bf08 R08: 0000000000000001 R09: 0000000000000000
[ 3358.674766] R10: 0000000000000000 R11: 0000000000000038 R12: 0000000000000000
[ 3358.674766] R13: 0000000000000000 R14: 00000027cd2c9801 R15: 0000000000000000
[ 3358.674766] df_debug+0x2d/0x30
[ 3358.674766] do_double_fault+0x9a/0x130
[ 3358.674766] double_fault+0x1e/0x30
[ 3358.674766] RIP: 0010:0x1520
[ 3358.674766] Code: Bad RIP value.
[ 3358.674766] RSP: 0018:0000000000007200 EFLAGS: 00010096
[ 3358.674766] RAX: 0000000000000102 RBX: 00000000f7b499e8 RCX: 00000000ffb06f74
[ 3358.674766] RDX: 000000000000002f RSI: 00000000f7b49980 RDI: 00000000f7f5c000
[ 3358.674766] RBP: 00000000ffb06bf0 R08: 0000000000000000 R09: 0000000000000000
[ 3358.674766] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 3358.674766] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 3358.674766] </#DF>
[ 3358.674766] ---[ end trace e3d2b86de2ecdfca ]---
[ 3358.674767] ------------[ cut here ]------------

Uldis (UD)
Posts: 5081
Joined: Wed Mar 15, 2017 4:44 pm
Location: London
Contact:

Re: Kernel Panic when running 2 Vendors at the same time

Post by Uldis (UD) » Fri Mar 20, 2020 3:21 pm

Azure is not in the EVE supported clouds
It is exactly because this non supported qemu and virtualization..
stop it
we do not support azure
https://www.eve-ng.net/index.php/docume ... quirement/

eboman
Posts: 3
Joined: Fri Mar 20, 2020 12:00 pm

Re: Kernel Panic when running 2 Vendors at the same time

Post by eboman » Fri Mar 20, 2020 4:41 pm

for what it is worth, seems when running vIOs images it does not occur, seems to be an combination between IOL and Qemu.

Uldis (UD)
Posts: 5081
Joined: Wed Mar 15, 2017 4:44 pm
Location: London
Contact:

Re: Kernel Panic when running 2 Vendors at the same time

Post by Uldis (UD) » Fri Mar 20, 2020 6:32 pm

it is just very bad kernel used in Azure
We do not support it.
on your own risk mate

Post Reply