1. Forum
  2. Usenet
  3. LINUX.DEBIAN.USER

  • QEMU: Run a container of a different architecture

    From Steve Keller@21:1/5 to All on Wed Aug 21 17:20:01 2024
    Can I run a container for a different CPU architecture using
    systemd-nspawn? I can easily install on my amd64 host a Debian
    container of the same architecture and run that:

    # debootstrap stable deb12-amd64
    # systemd-nspawn -D deb12-amd64

    and get a shell running in that container. I can also install a
    Debian system of a different architecture and run binaries from it
    like this (using qemu-user-binfmt):

    # debootstrap --arch=arm64 --foreign stable deb-arm64
    # QEMU_LD_PREFIX=/usr/aarch64-linux-gnu deb-arm64/bin/date
    Wed Aug 21 16:43:40 CEST 2024

    But the following doesn't work

    # QEMU_LD_PREFIX=/usr/aarch64-linux-gnu systemd-nspawn -D deb-arm64
    Spawning container deb-arm64 on /usr/local/deb-arm64.
    Press ^] three times within 1s to kill container.
    execv(/bin/bash, /bin/bash, /bin/sh) failed: No such file or directory
    Container deb-arm64 failed with error code 1.

    I'd like to know if it's also possible to run the whole container in
    arm64 architecture using systemd-nspawn like above for amd64.

    Steve

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?Q?Markus_Sch=C3=B6nhaber?=@21:1/5 to All on Wed Aug 21 19:10:01 2024
    21.08.24, 16:56 +0200, Steve Keller:

    Can I run a container for a different CPU architecture using
    systemd-nspawn?

    No. systemd-nspawn does indeed simply run a container. You can think of
    that as a chroot on steroids. This means, everything inside the
    container is run using the host kernel. Or to put it the other way
    round: what the host kernel can't execute won't run.

    If you want to run stuff for an architecture the host kernel cannot cope
    with, you need a virtual machine (manager like QEMU). You can run one
    using something like
    qemu-system-aarch64 [...]
    but it's probably much easier to use some helper like libvirt.

    I can easily install on my amd64 host a Debian
    container of the same architecture and run that:

    # debootstrap stable deb12-amd64
    # systemd-nspawn -D deb12-amd64

    and get a shell running in that container. I can also install a
    Debian system of a different architecture and run binaries from it
    like this (using qemu-user-binfmt):

    # debootstrap --arch=arm64 --foreign stable deb-arm64
    # QEMU_LD_PREFIX=/usr/aarch64-linux-gnu deb-arm64/bin/date
    Wed Aug 21 16:43:40 CEST 2024

    But the following doesn't work

    # QEMU_LD_PREFIX=/usr/aarch64-linux-gnu systemd-nspawn -D deb-arm64
    Spawning container deb-arm64 on /usr/local/deb-arm64.
    Press ^] three times within 1s to kill container.
    execv(/bin/bash, /bin/bash, /bin/sh) failed: No such file or directory
    Container deb-arm64 failed with error code 1.

    I'd like to know if it's also possible to run the whole container in
    arm64 architecture using systemd-nspawn like above for amd64.
    As said above: no.

    --
    Regards
    mks

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Steve Keller@21:1/5 to All on Wed Aug 21 22:30:01 2024
    "Markus Schönhaber" writes:

    No. systemd-nspawn does indeed simply run a container. You can think of
    that as a chroot on steroids. This means, everything inside the
    container is run using the host kernel. Or to put it the other way
    round: what the host kernel can't execute won't run.

    Actually, the kernel can execute such foreign ELF binaries. This
    works with the binfmt_misc file system and registering the appropriate interpreter which is then automaically executed by the kernel. This is
    similar to the she-bang (!#) mechanism where the kernel also loads the
    needed interpreter.

    I've shown that in my mail with executing the aarch64 date binary
    directly on the amd64 kernel. I didn't see any reason why this
    shouldn't work with all binaries in a container. Now, when looking at
    it again, I think the problem is the missing interpreter in the
    container's file system. Quite obvious. OK, I'll try copying the
    QEMU interpreter qemu-aarch64 and all needed libs and other files into
    the appropriate place...

    Steve

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Tim Woodall@21:1/5 to Steve Keller on Thu Aug 22 19:30:01 2024
    On Wed, 21 Aug 2024, Steve Keller wrote:

    Can I run a container for a different CPU architecture using
    systemd-nspawn? I can easily install on my amd64 host a Debian
    container of the same architecture and run that:


    Don't know about systemd-nspawn but I do something like this using
    unshare, binfmt-support and qemu-user-static.

    I don't have to do anything at all other than create the file system
    with the emulated architecture and then chroot into it with those
    packages installed.

    $ apt-mark showmanual | grep binf
    binfmt-support
    $ apt-mark showmanual | grep qemu
    qemu-user-static


    I don't think I installed anything else but I might have missed
    something.

    Perhaps systemd-nspawn would similarly work with those packages
    installed.

    Tim.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Steve Keller@21:1/5 to Tim Woodall on Fri Aug 23 16:30:01 2024
    "Tim Woodall" <debianuser@woodall.me.uk> writes:

    Don't know about systemd-nspawn but I do something like this using
    unshare, binfmt-support and qemu-user-static.

    I don't have to do anything at all other than create the file system
    with the emulated architecture and then chroot into it with those
    packages installed.

    Thanks, the hint to qemu-user-static solved the issue very easily.
    Nothing to install inside the container or other things to setup. I
    just installed qemu-user-static, which removed qemu-user-binfmt
    because qemu-user-static brings its own binfmt support.

    After installing I could simply run systemd-nspawn -D deb-arm64 and it
    worked immediately, where deb-arm64 is the directory to which I have
    installed the Debian/arm64 system using debootstrap.

    I think systemd-nspawn does something very similar to what you do with
    unshare and chroot.

    Perhaps systemd-nspawn would similarly work with those packages
    installed.

    Yes, out of the box. I find this a litte bit surprising since

    # head -n2 /proc/sys/fs/binfmt_misc/qemu-aarch64
    enabled
    interpreter /usr/libexec/qemu-binfmt/aarch64-binfmt-P
    # ls -l /usr/libexec/qemu-binfmt/aarch64-binfmt-P
    lrwxrwxrwx 1 root root 29 May 9 07:44 /usr/libexec/qemu-binfmt/aarch64-binfmt-P -> ../../bin/qemu-aarch64-static

    these paths are not visible in the chroot environment or the mount
    namespace, respectively. That means the kernel probably does the
    interpreter lookup earlier, not at execution time in the chrooted
    environment. Besides that, qemu-aarch64-static doesn't need any other
    files from the host environment. In contrast, qemu-aarch64 which I had
    used instead, needs libraries which it cannot find in the chrooted
    environment and therefore fails.

    Nevertheless, it would be nice to be able to run qemu-aarch64
    dynamically linked which finds its libs in the host environment but
    runs the command in the current inherited environment. But I guess
    that's impossible.

    Steve

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)