QEMU: Run a container of a different architecture

[Steve Keller]

Can I run a container for a different CPU architecture using
systemd-nspawn? I can easily install on my amd64 host a Debian
container of the same architecture and run that:

# debootstrap stable deb12-amd64
# systemd-nspawn -D deb12-amd64

and get a shell running in that container. I can also install a
Debian system of a different architecture and run binaries from it
like this (using qemu-user-binfmt):

# debootstrap --arch=arm64 --foreign stable deb-arm64
# QEMU_LD_PREFIX=/usr/aarch64-linux-gnu deb-arm64/bin/date
Wed Aug 21 16:43:40 CEST 2024

But the following doesn't work

# QEMU_LD_PREFIX=/usr/aarch64-linux-gnu systemd-nspawn -D deb-arm64
Spawning container deb-arm64 on /usr/local/deb-arm64.
Press ^] three times within 1s to kill container.
execv(/bin/bash, /bin/bash, /bin/sh) failed: No such file or directory
Container deb-arm64 failed with error code 1.

I'd like to know if it's also possible to run the whole container in
arm64 architecture using systemd-nspawn like above for amd64.

Steve

[Markus Schönhaber]

Post by Steve Keller
Can I run a container for a different CPU architecture using
systemd-nspawn?

No. systemd-nspawn does indeed simply run a container. You can think of
that as a chroot on steroids. This means, everything inside the
container is run using the host kernel. Or to put it the other way
round: what the host kernel can't execute won't run.

If you want to run stuff for an architecture the host kernel cannot cope
with, you need a virtual machine (manager like QEMU). You can run one
using something like
qemu-system-aarch64 [...]
but it's probably much easier to use some helper like libvirt.

Post by Steve Keller
I can easily install on my amd64 host a Debian
# debootstrap stable deb12-amd64
# systemd-nspawn -D deb12-amd64
and get a shell running in that container. I can also install a
Debian system of a different architecture and run binaries from it
# debootstrap --arch=arm64 --foreign stable deb-arm64
# QEMU_LD_PREFIX=/usr/aarch64-linux-gnu deb-arm64/bin/date
Wed Aug 21 16:43:40 CEST 2024
But the following doesn't work
# QEMU_LD_PREFIX=/usr/aarch64-linux-gnu systemd-nspawn -D deb-arm64
Spawning container deb-arm64 on /usr/local/deb-arm64.
Press ^] three times within 1s to kill container.
execv(/bin/bash, /bin/bash, /bin/sh) failed: No such file or directory
Container deb-arm64 failed with error code 1.
I'd like to know if it's also possible to run the whole container in
arm64 architecture using systemd-nspawn like above for amd64.

As said above: no.

[Steve Keller]

Post by Markus Schönhaber
No. systemd-nspawn does indeed simply run a container. You can think of
that as a chroot on steroids. This means, everything inside the
container is run using the host kernel. Or to put it the other way
round: what the host kernel can't execute won't run.

Actually, the kernel can execute such foreign ELF binaries. This
works with the binfmt_misc file system and registering the appropriate
interpreter which is then automaically executed by the kernel. This is
similar to the she-bang (!#) mechanism where the kernel also loads the
needed interpreter.

I've shown that in my mail with executing the aarch64 date binary
directly on the amd64 kernel. I didn't see any reason why this
shouldn't work with all binaries in a container. Now, when looking at
it again, I think the problem is the missing interpreter in the
container's file system. Quite obvious. OK, I'll try copying the
QEMU interpreter qemu-aarch64 and all needed libs and other files into
the appropriate place...

Steve

[Tim Woodall]

Post by Steve Keller
Can I run a container for a different CPU architecture using
systemd-nspawn? I can easily install on my amd64 host a Debian

Don't know about systemd-nspawn but I do something like this using
unshare, binfmt-support and qemu-user-static.

I don't have to do anything at all other than create the file system
with the emulated architecture and then chroot into it with those
packages installed.

$ apt-mark showmanual | grep binf
binfmt-support
$ apt-mark showmanual | grep qemu
qemu-user-static


I don't think I installed anything else but I might have missed
something.

Perhaps systemd-nspawn would similarly work with those packages
installed.

Tim.

[Steve Keller]

Post by Tim Woodall
Don't know about systemd-nspawn but I do something like this using
unshare, binfmt-support and qemu-user-static.
I don't have to do anything at all other than create the file system
with the emulated architecture and then chroot into it with those
packages installed.

Thanks, the hint to qemu-user-static solved the issue very easily.
Nothing to install inside the container or other things to setup. I
just installed qemu-user-static, which removed qemu-user-binfmt
because qemu-user-static brings its own binfmt support.

After installing I could simply run systemd-nspawn -D deb-arm64 and it
worked immediately, where deb-arm64 is the directory to which I have
installed the Debian/arm64 system using debootstrap.

I think systemd-nspawn does something very similar to what you do with
unshare and chroot.

Post by Tim Woodall
Perhaps systemd-nspawn would similarly work with those packages
installed.

Yes, out of the box. I find this a litte bit surprising since

# head -n2 /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /usr/libexec/qemu-binfmt/aarch64-binfmt-P
# ls -l /usr/libexec/qemu-binfmt/aarch64-binfmt-P
lrwxrwxrwx 1 root root 29 May 9 07:44 /usr/libexec/qemu-binfmt/aarch64-binfmt-P -> ../../bin/qemu-aarch64-static

these paths are not visible in the chroot environment or the mount
namespace, respectively. That means the kernel probably does the
interpreter lookup earlier, not at execution time in the chrooted
environment. Besides that, qemu-aarch64-static doesn't need any other
files from the host environment. In contrast, qemu-aarch64 which I had
used instead, needs libraries which it cannot find in the chrooted
environment and therefore fails.

Nevertheless, it would be nice to be able to run qemu-aarch64
dynamically linked which finds its libs in the host environment but
runs the command in the current inherited environment. But I guess
that's impossible.

Steve