We recently noticed an issue with an Azure kernel that is included by default with, for example, a brand-new Ubuntu Server LTS VM image. Surprising.
The symptoms: a server (VM) becomes partially unreachable at certain moments. This can manifest itself in the server no longer being accessible through its domain name, but also through SSH.
The server does not crash completely. The Azure Portal also shows that the VM is still running. Yet one or more vhosts can become unreachable, as can SSH. The VM therefore appears to be active, but in practice it can no longer be considered reliable.
The logs showed that something was going wrong at the kernel level. One of the messages was:
kernel: #PF: error_code(0x0000) - not-present page Upon further investigation, one particular line stood out:
RIP: strlen+0x0/0x30 Combined with the other messages, the following picture emerged:
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
CR2: 0000000100000000 But why is the kernel trying to read a memory address that is not available? It turned out to be an interesting combination of circumstances.
The logs also contained the following messages:
Data path switched from VF: enP28525s1
VF unregistering: enP28525s1
BUG: unable to handle page fault
Workqueue: hv_pci_6f6d hv_eject_device_work
note: kworker/u64:0 exited with irqs disabled In this particular case, this turned out to indicate that a Mellanox Virtual Function (VF) was being disconnected and removed from the kernel. This happens through the Azure PCI mechanism, where the hv_pci workqueue handles the removal.
Our suspicion is that this is related to maintenance activities on the Azure infrastructure, for example maintenance or a live migration during which the virtual network interface is changed. In any case, the logs show that the VF is unregistered, after which a kernel page fault occurs while processing that removal.
The page fault occurs during a read operation in kernel mode, with execution reaching strlen. Apparently, an invalid memory address is being accessed in the process. The result is a kernel error in the worker handling the removal of the PCI device. The fact that the worker exits with interrupts disabled makes the situation even more concerning.
The consequences?
The result is that the VM does not necessarily crash completely, but can still lose essential functionality. Needless to say, this is unacceptable in a production environment.
It turns out we are not the only ones who have encountered this issue. Reports can also be found on the Ubuntu bug tracker and mailing lists:
https://mail-archive.com/ubuntu-bugs%40lists.ubuntu.com/msg6329890.html
https://mid.mail-archive.com/ubuntu-bugs%40lists.ubuntu.com/msg6327362.html
But it does make you wonder how a large organisation like Microsoft manages to ship an Azure kernel in a VM image that, under certain circumstances, causes a kernel error while handling a change to the virtual hardware. Potentially leaving the server unusable for the end customer.
In this case, everything on the server is up to date, including the operating system. However, there is currently no newer kernel available through the Azure kernel updates. linux-azure 7.0.0-1017 is the latest version available for this configuration.
This means that the issue is still present in this configuration at the moment, and that a regular update does not provide a solution.
The temporary solution
For now, the apparent solution is to disable Accelerated Networking. This prevents the VM from using the relevant SR-IOV networking functionality and its associated Virtual Functions.
After that, all we can do is wait for a kernel patch that actually fixes the problem. Once one becomes available, Accelerated Networking can potentially be enabled again. Naturally, it would be wise to test this in a controlled manner first.
It remains remarkable that an issue like this made it through QA, especially considering that this involves an Azure configuration provided by Microsoft in combination with a popular Ubuntu Server image.
We would have liked to send Microsoft an email about this, but Microsoft 364 was down. Sorry. Bad joke.