• Re: Trixie: cryptsetup hangs at boot when trying to create a device fil

    From Guilhem Moulin@21:1/5 to Konstantin Khomoutov on Wed Jul 2 01:10:01 2025
    Hi,

    On Wed, 02 Jul 2025 at 01:02:02 +0300, Konstantin Khomoutov wrote:
    I've hit a regression after upgrading my Debian 12 laptop to Trixie -
    in short, cryptsetup hangs at *early* boot stage when it apparently trying to create a device mapper node for a LUKS1 device it successfully opened.
    LUKS2 devices behave just fine.
    […]
    2. Then the boot process hangs for a while and then fails with the message telling it failed to bring up the /boot partition and hence the local-fs.target failed to be brought up, as well - bailing into the
    emergency mode (which was unusable as I had root account with the locked password, but this is another story).
    […]
    The most weird thing is that if I comment out a line for that /boot
    partition in /etc/fstab so that it's not attempted to be brought up at boot and then manually open the device using cryptsetup after the system as
    booted - it gets opened just fine.

    AFAICT it doesn't hang at early boot (initramfs) stage, but rather after
    the system has been handed over to systemd. Do you have
    systemd-cryptsetup installed? (See the NEWS entry for systemd=256-2,
    which I guess should be in the release notes too.)

    1. https://cryptsetup-team.pages.debian.net/cryptsetup/encrypted-boot.html

    I need to review and update this page GRUB can open LUKS2 devices these
    days. And systemd-cryptsetup should be mentioned there too indeed.

    --
    Guilhem.

    -----BEGIN PGP SIGNATURE-----

    iQIzBAEBCgAdFiEERpy6p3b9sfzUdbME05pJnDwhpVIFAmhkaJcACgkQ05pJnDwh pVJy2BAAlxK2M/syODUKraEJFOMLOCTToB+LZuK7CRNjmO2LCiJjRnbWXo0MIZ2j LYFPZ0uaXv+b4CBeF3fRVbilv52wO+GCc6VWlbaYrJQyr0XLHAkljpNQKMLHt5RN xErcoLS1/fpYewNf96lyUfip0SxiDTgtUY9u0P5Qj+05JP8BEZdx46vc1egcRchF YoYi8hUD7qiZzeRfqXVI3jB+xXdxAvJWGE06AvSc2ipffg2f+pBLOSUtizi3n8Pd 6D2vyYKdwnjVgFSPrA2qNkIkDUVpVhNodwLAG0itgTjRh6u1I21N1Ev4ZFaEVK5P tmYIB8EJh0pbADtp001GzUaGfTjUZBfIdF6Psemqcdo0113Ch7eredvqYCDqd57s dr6RGa3yMeGh1qIqm3X7tH8V77zfNCmWPzUMAI38ZTBaQkfdu5GpUeLSFLr1OoVh 7LVkZ7TuJfmKUZlwwBt+QszGgPL6XjYiRmKrpRj4b3yt9OL/MErMG2e/Gg1f0okg QmaQuasfCLJwyv9WQJDqMpzV5p5e/wrsgYVzH8eTG7K6ZzY8oUxVmg0mMzQkwmX1 iP0p5VHZRK675YB0pxUJQaFX5lNH6nfC0HaVFnwIQu7rfXOD9t8tr0in3JNCEHAo ROh1a63wCBnP/R3djdvltmtJVG7LeVgWLjo4A961q2nvyqeOwX4=
    =YaVX
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Konstantin Khomoutov@21:1/5 to Guilhem Moulin on Wed Jul 2 12:40:01 2025
    On Wed, Jul 02, 2025 at 01:00:39AM +0200, Guilhem Moulin wrote:

    [...]
    I've hit a regression after upgrading my Debian 12 laptop to Trixie -
    in short, cryptsetup hangs at *early* boot stage when it apparently trying to
    create a device mapper node for a LUKS1 device it successfully opened. LUKS2 devices behave just fine.
    […]
    2. Then the boot process hangs for a while and then fails with the message telling it failed to bring up the /boot partition and hence the local-fs.target failed to be brought up, as well - bailing into the emergency mode (which was unusable as I had root account with the locked password, but this is another story).
    […]
    The most weird thing is that if I comment out a line for that /boot partition in /etc/fstab so that it's not attempted to be brought up at boot and then manually open the device using cryptsetup after the system as booted - it gets opened just fine.

    AFAICT it doesn't hang at early boot (initramfs) stage, but rather after
    the system has been handed over to systemd.

    I don't think so. I think I've failed to properly communicate the crucial
    point of my story ;-) Let me reiterate:

    I've noticed that the boot process brings up the LUKS2-encrypted root parition just fine

    In my original mail I've mistakenly said "boot" there ^, but in fact it was
    the root partition.

    and tried to debug the boot process by booting the kernel with the

    root=/dev/mapper/root-crypt init=/usr/bin/bash

    arguments and then trying to open the LUKS1-encrypted /boot by hand.

    Here is what happens:

    - cryptsetup is able to verify the password used for one of the key slots
    just fine.

    - But when I run it to actually open the device it

    1. Successfully opens the device and creates a node available via
    dmsetup (sorry, I don't know the correct terminology - I mean, one
    can see the new device by running `dmsetup ls` and mount it as
    /dev/dm-2).

    2. It then hangs dead - apparently when attempting to create a node
    under /dev/mapper.

    So, the kernel gets booted with /usr/bin/bash as its init process, and so I assume there are no parts of systemd running - except maybe something from initrafms (?). Still, in this scenario, the LUKS2-formatted root partition
    gets brought up just fine and has a node under /dev/mapper created for it just fine (and it allows the root= kernel argument to work in my case), but a
    manual attempt to bring up the LUKS1-formatted /boot partition hangs.
    The call I've did is

    # cryptsetup open --type=luks1 --disable-locks --disable-keyring \
    --key-slot=0 /dev/nvme0n1p2 boot-crypt

    Actually all the options make no difference - I've tried different
    combinations of them - the program hangs at exactly the same syscall
    after the device is opened, it can be seen in the output of

    # dmsetup ls

    but it still has no node under /dev/mapper.

    Hence I'm inclined to think it's still an "early boot" issue (however moot
    this term is).

    Do you have systemd-cryptsetup installed? (See the NEWS entry for systemd=256-2, which I guess should be in the release notes too.)

    Thanks, I've try to research this ASAP.
    I think it would be interesting to record what processes are running when
    that "bash-as-init-process" get started.

    1. https://cryptsetup-team.pages.debian.net/cryptsetup/encrypted-boot.html

    I need to review and update this page

    Oh, I've now noticed you're the author of this manual.
    That's a good manual indeed, kudos for this work!

    GRUB can open LUKS2 devices these days.

    I've did a quick research of this topic after I've hit the regression we're discussing, and it seems that the support is still incomplete [2]. In particular, it appears one has to use PBKDF for key material derived from passwords as the version of libgcrypt used by GRUB does not support argon2id which seems to be the default for this in cryptsetup for LUKS2. IOW, extra
    care must be taken by the user when setting up a LUKS2-formatted /boot.

    2. https://savannah.gnu.org/bugs/?55093

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Guilhem Moulin@21:1/5 to Konstantin Khomoutov on Wed Jul 2 13:20:01 2025
    On Wed, 02 Jul 2025 at 13:13:39 +0300, Konstantin Khomoutov wrote:
    On Wed, Jul 02, 2025 at 01:00:39AM +0200, Guilhem Moulin wrote:
    I've noticed that the boot process brings up the LUKS2-encrypted root
    parition just fine

    In my original mail I've mistakenly said "boot" there ^, but in fact it was the root partition.

    Devices holding the root FS and the resume device are unlocked at early
    boot stage. Those holding /boot are normally not (because /boot is only
    needed later in the boot process), and instead unlocked by systemd. I
    believe LUKS1 vs LUKS2 is a coincidence here; what matters instead is at
    which stage each device is unlocked.

    and tried to debug the boot process by booting the kernel with the

    root=/dev/mapper/root-crypt init=/usr/bin/bash

    arguments and then trying to open the LUKS1-encrypted /boot by hand.

    Here is what happens:

    - cryptsetup is able to verify the password used for one of the key slots
    just fine.

    - But when I run it to actually open the device it

    1. Successfully opens the device and creates a node available via
    dmsetup (sorry, I don't know the correct terminology - I mean, one
    can see the new device by running `dmsetup ls` and mount it as
    /dev/dm-2).

    2. It then hangs dead - apparently when attempting to create a node
    under /dev/mapper.

    So, the kernel gets booted with /usr/bin/bash as its init process, and so I assume there are no parts of systemd running - except maybe something from initrafms (?).

    Since you get to to the point where execution is handed over to init=/usr/bin/bash
    or init=/sbin/init it is *not* an early boot issue.

    Still, in this scenario, the LUKS2-formatted root partition
    gets brought up just fine and has a node under /dev/mapper created for it just
    fine (and it allows the root= kernel argument to work in my case), but a manual attempt to bring up the LUKS1-formatted /boot partition hangs.

    Probably not for the same reason though (and FWIW assuming this is a
    production system, this was perhaps not a good idea to attach that
    strace output to a public mailing list). I'm not sure how init=/usr/bin/bash interacts with udev, but cryptsetup seems to hang while waiting for the
    udev cookie (`cryptsetup --debug` output would have been more useful and
    less risky than a strace output). Does it still hang when you set the environment variable `DM_DISABLE_UDEV=y`?

    Either way, experimenting with init=/usr/bin/bash and the cryptsetup(8) executable is not a reproducer for your issue since because you end up
    in a very different environment. I think systemd doesn't hang while
    waiting for the udev cookie, but instead fails because it lacks
    cryptsetup integration or another unrelated reason.

    Hence I'm inclined to think it's still an "early boot" issue

    No it's not.

    Do you have systemd-cryptsetup installed? (See the NEWS entry for
    systemd=256-2, which I guess should be in the release notes too.)

    You need to install the systemd-cryptsetup binary package to let systemd automatically unlock your boot device. See #1079644 or #1076208.

    If you have systemd-cryptsetup installed, then increasing the log level
    might shed some light about what is going on, see https://freedesktop.org/wiki/Software/systemd/Debugging/ .

    --
    Guilhem.

    -----BEGIN PGP SIGNATURE-----

    iQIzBAEBCgAdFiEERpy6p3b9sfzUdbME05pJnDwhpVIFAmhlE5YACgkQ05pJnDwh pVLY6Q/9GVSGrvdP5hdPjwCRh7q2Qr/Obb5XHr0sS7a1Gynu4fnYlXYr1LNnrUBZ JN6tFjGKBXFqvsmhtmBGF+aK5He83K9sLRDjLDWwR40SpOSbFZIo96MZxYOF/LKm YQgADspqJpIafbT7RfJm2vgMpO6aY8zWQ6lNJgkxEUHtBTsozZakAIFAGSy4oQWx tTOAvqkOCoTQ3LVn+9ACpFZMqdSeurLBiYRVP0ikS4bSjTugIClP+Dn8KSr+gJa8 VM+RSpvih4G3LL+PUkdfxWwbcs15zF+MJraIrPiRSn/50n15IaHqfUoYnq9mpRsK THGHacA6fmlkB3O528zYdkzbJy34Ze2xtTm9HrWiWjnmVi2fKtakIU1qRRyCapLe myeWWLxOcB3vTJe3tFdozub+i5qxcqad+y1PNDAfMHoJVz9JJbcKKHqdGV+jnP4W fvwp+X5I5m45HkY15F3E5cTdJkKxjWu8XAuWo5xuMIVTTKUcvAKlxmCsSSf7vEAl Wr6DB7fDztCZ5r30BPULbH2/BuZR/YQNzF7fn5F586IqgSQTivp+29IeGk4KNZ/e 84l2Y6Hu6YPBxPzfMAcKDaZIXT/yJ9ZpZ0dwYh1Q3zBExnegTOmdJguw5CLFuB45 x5bGAQ47r1wfjrz9DCuszsYxGX2+WqrW9taLVz5rzXIspl6D544=
    =x/Hq
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Konstantin Khomoutov@1:229/2 to Guilhem Moulin on Thu Jul 3 00:00:02 2025
    From: kostix@bswap.ru

    On Wed, Jul 02, 2025 at 01:10:16PM +0200, Guilhem Moulin wrote:

    [...]
    Hence I'm inclined to think it's still an "early boot" issue

    No it's not.

    Do you have systemd-cryptsetup installed? (See the NEWS entry for
    systemd=256-2, which I guess should be in the release notes too.)

    You need to install the systemd-cryptsetup binary package to let systemd automatically unlock your boot device. See #1079644 or #1076208.

    Yes, your guess was a spot-on: installing cryptsetup-system fixed the issue completely for me.

    Looks like I'm going to propose an addition to the release notes as folks in the bugs you mentioned basically have hit the same problem.

    Thanks for helping me out, Guilhem!

    --- SoupGate-Win32 v1.05
    * Origin: you cannot sedate... all the things you hate (1:229/2)
  • From Konstantin Khomoutov@21:1/5 to Konstantin Khomoutov on Sat Jul 5 18:40:02 2025
    On Thu, Jul 03, 2025 at 12:38:32AM +0300, Konstantin Khomoutov wrote:

    [...]
    Do you have systemd-cryptsetup installed? (See the NEWS entry for
    systemd=256-2, which I guess should be in the release notes too.)

    You need to install the systemd-cryptsetup binary package to let systemd
    automatically unlock your boot device. See #1079644 or #1076208.

    Yes, your guess was a spot-on: installing cryptsetup-system fixed the issue completely for me.

    Looks like I'm going to propose an addition to the release notes as folks in the bugs you mentioned basically have hit the same problem.

    Filed #1108823 with a proposed patch to the release notes.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)