foitzik.dev // my collection of random things and thoughts

Minisforum R1 aarch64 VM weirdness

tags:

RAM is expensive. Therefore I bought a Minisforum R1 to add as additional lab station. Did some research on it. It uses a custom kernel. Welp. Should not affect VMs in PXVirt that much I thought.

Weeks later, when upgrading to nixOS 26.05 on my test VMs I started seeing extremely odd things.

The setup is mostly automated using various things like tofu etc.

Building VMs is based on a topology and nix builds them and the helper scripts for it including the tofu/terraform to deploy them incl. MAC based IP/Hostname assignment with LXC dnsmasq container.

==Disclaimer: I am not an expert w.r.t. to virtualization/the kernel, so if this blog posts contains any inaccuracies PLEASE reach out to me and correct me. I am happy to receive feedback and learn something.==

The symptom

The VM started normally and displayed the PXVirt splash screen. The bootloader appeared, showed the nixOS boot entries and selected one.

Immediately after the bootloader handed control to Linux, noVNC displayed:

Display output is not active

This alone would not have worried me too much. Losing graphical output after the kernel switches consoles is not exactly unheard of.

Unfortunately, the VM also never appeared on the network:

$ ssh -A -J [email protected] [email protected]
ssh: connect to host 10.xx.0.xx port xxxx: Connection timed out

No DHCP activity. No SSH. No serial login. Rebooting did not help.

With nixOS 25.11 the problem had appeared intermittently. Some boots worked, while others died after selecting the boot entry. With 26.05 it became effectively reproducible.

There were a few possible initial culprits since it seemd to happen right after selecting a boot entry where either component was corrupted or working incorrectly:

It was not the bootloader

But finding that out was also "hard", because it was flaky as hell and I switched between 25.11 and 26.05 sometimes losing track which version I used.

Trying to get output from a dead guest

The VM already had a serial device configured in PXVirt:

serial_device = {
  device = "socket";
};

The guest kernel was also configured for the PL011 serial port provided by QEMU's aarch64 virt machine:

boot.kernelParams = [
  "console=tty0"
  "console=ttyAMA0,115200n8"
  "loglevel=7"
  "systemd.log_level=debug"
  "systemd.show_status=1"
  "rd.systemd.show_status=1"
  "panic=1"
  "reboot=t"
];

I also explicitly included the relevant virtio modules in the initrd:

boot.initrd.availableKernelModules = [
  "virtio_pci"
  "virtio_scsi"
  "virtio_blk"
  "virtio_net"
  "sr_mod"
  "virtio_gpu"
];

Still, qm terminal 401 remained empty after the bootloader. I spent so much time thinking it was a serial / display issue. Which MADE NO SENSE because I was also always testing ssh / ping etc. and those were also not working.

At this point I gathered all my findings and let GPT 5.5 Pro and Opus 4.8 go through the host while the VM was booting because I was very much out of my depth.

The clue in the PXVirt host log

While starting the VM, the PXVirt host kernel logged this:

kvm [2868]: Unsupported guest sys_reg access at: ffffc13ae635b6b0 [000003c5]
{ Op0(3), Op1(0), CRn(10), CRm(4), Op2(4), func_read },

The QEMU process also consumed an entire CPU core while the guest made no progress.

ARM system registers are identified through the Op0, Op1, CRn, CRm and Op2 fields. The register encoding in the error was:

S3_0_C10_C4_4

That maps to:

MPAMIDR_EL1

I had to look up what MPAM is: Apparently it stands for Memory Partitioning and Monitoring. It is an ARM feature intended to partition and monitor shared memory-system resources.

What actually happened

The PXVirt VM used QEMU's virt machine type with OVMF and:

-cpu host

The host therefore exposed the physical CPU's feature information to the guest.

The guest saw MPAM advertised through the ARM CPU identification registers. The newer nixOS 26.05 guest kernel, Linux 6.18.34 in this case, detected the feature and tried to read MPAMIDR_EL1.

But the custom host kernel used by the Minisforum/PXVirt installation did not correctly virtualize that register.

The resulting sequence was approximately:

  1. The virtual CPU tells the guest that MPAM exists.
  2. The guest kernel believes it.
  3. The guest reads an MPAM system register during early boot.
  4. KVM does not know how to handle the access.
  5. KVM injects an undefined instruction into the guest.
  6. The guest dies before networking, SSH or a useful console is available.
  7. noVNC says that display output is inactive, which sends me investigating bootloaders and GPUs.

In other words, the virtual CPU advertised a feature which the virtualization stack could not actually provide.

I did not do a deepdive whether 25.11 is unaffected and whether this was introduced with 26.05.

The one-line workaround

The Linux kernel has a command-line option to unconditionally disable MPAM on arm64:

arm64.nompam

Adding it to the base VM configuration was rather unspectacular:

boot.kernelParams = [
  "console=tty0"
  "console=ttyAMA0,115200n8"
  "loglevel=7"
  "systemd.log_level=debug"
  "systemd.show_status=1"
  "rd.systemd.show_status=1"
  "panic=1"
  "reboot=t"

  # The host exposes MPAM to the guest but does not correctly virtualize
  # the MPAM system registers. Disable MPAM before the guest probes them.
  "arm64.nompam"
];

Now I just rebuilt the image, deployed it and instructed an agent to basically reboot the machine like 5-10 times and check for any kernel error logs while I was doing quests in the Swamp Camp, trying to impress Cor Calom.

Spoiler none appeared.

And it also made the VM responsive again. Previously it was not properly responding to regular shutdown/reboot commands and had to be hard reset'd.

Is this the proper fix?

No. But at this point and for a test server I do not care to spent more time on it, even though I used GPT 5.5 Pro and Opus 4.8 hard in the final phase to bruteforce the root cause detection after I narrowed it down to bootloader linux kernel handover.

Outlook

I am not exactly sure how one would fix this, but this is so far away from what I wanted to do that at this point in time: It's not a priority. If Minisforum wants to pay me to mainline their stuff and fix things like this, they should pay me. But they never reached out to me. I wrote them before encountering this issue to talk about some sort of stipend/sponsorship to get nixOS working (properly) on their hardware.

Published 2026-06-13
home ↩