• Re: Kernel 6.9.9 (amd64) results in huge initrd / initramfs size

    From Dan Ritter@21:1/5 to Celejar on Thu Jul 18 17:20:01 2024
    Celejar wrote:
    Hello,

    I'm currently on kernel 6.9.8 (amd64 / Sid). Installing 6.9.9 fails due to running out of space on /boot:

    ...

    I'm not sure why I'm hitting this now - did Debian just change
    something? Is anyone else hitting this? Is this documented somewhere?
    Is there a straightforward fix / workaround?

    Of course something changed: it's Sid.

    It will probably be straightened out before it hits stable.

    Do you need this firmware? If so, do you need it at boot time?
    There are kernel build options. Building it as a module and
    making sure that initrd doesn't include it is pretty normal for
    a number of kernel modules.

    -dsr-

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andy Smith@21:1/5 to Celejar on Thu Jul 18 17:50:02 2024
    Hi,

    On Thu, Jul 18, 2024 at 11:35:15AM -0400, Celejar wrote:
    I'd rather not mess around with stuff I don't really understand
    without an official guide to the process.

    I don't mean this to be snarky, but that desire seems incompatible
    with running Debian sid. I honestly think it's an unreasonable
    expectation to want official guides for every transitory broken
    state in a development tree.

    Asking if a specific thing is a known issue that is being worked
    on? Sure. Expecting it to be documented before any user hits it? Not
    so much.

    Thanks,
    Andy

    --
    https://bitfolk.com/ -- No-nonsense VPS hosting

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andrew M.A. Cater@21:1/5 to Andy Smith on Thu Jul 18 19:30:01 2024
    On Thu, Jul 18, 2024 at 03:42:30PM +0000, Andy Smith wrote:
    Hi,

    On Thu, Jul 18, 2024 at 11:35:15AM -0400, Celejar wrote:
    I'd rather not mess around with stuff I don't really understand
    without an official guide to the process.

    I don't mean this to be snarky, but that desire seems incompatible
    with running Debian sid. I honestly think it's an unreasonable
    expectation to want official guides for every transitory broken
    state in a development tree.


    If you run Sid, you are *expected* to be able to solve all your own
    problems. There are no guarantees other than if it breaks, you get
    to keep both pieces :)

    I recognise your name: you are an experienced user, but if you're not
    ready for speed of change / likelihood of serious breakage, please *don't*
    run Sid.

    The pace of change is high - apart from the freeze period prior to a major release, the likelihood of random breakage is very high. If there is a major transition (versions of GNOME, Qt versions for KDE, usrmerge for example)
    you could run into breakages that will last a few weeks.

    This is not the place for debugging Sid, I'm afraid, there are too few
    of us here that would run Sid normally.

    All the very best, as ever,

    Andy
    (amacater@debian.org)


    Asking if a specific thing is a known issue that is being worked
    on? Sure. Expecting it to be documented before any user hits it? Not
    so much.

    Thanks,
    Andy

    --
    https://bitfolk.com/ -- No-nonsense VPS hosting


    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Joe@21:1/5 to Greg Wooledge on Thu Jul 18 20:40:01 2024
    On Thu, 18 Jul 2024 14:10:21 -0400
    Greg Wooledge <greg@wooledge.org> wrote:

    On Thu, Jul 18, 2024 at 13:50:21 -0400, Celejar wrote:
    Really? I had the impression that lots of list subscribers / readers
    run Sid. Are there statistics on this?

    Nah, sid users are just louder, on average. Stable users don't have
    as much to talk about, because our stuff just works. ;-)


    ... thanks to many sid users helping with bug reports, and, indeed just
    using stuff to demonstrate that it isn't going to break instantly.

    And unless you're an experienced C/C++ programmer, then the correct way
    to deal with many/most sid breakages *is* to just wait, to stick with a previous version of something. Sometimes a bit of individual upgrading
    of things in a particular order can get things to install that the apt
    tools haven't managed in bulk. Certainly having more than one computer
    can help.

    I've used sid on my main workstation (not my server) since about the
    time of etch, and submitted a few bug reports over the years. And yes,
    I have a couple of spare computers which serve as backups if something
    vitally important breaks badly, but this is fairly rare. I did lose
    gEDA pcb for a while, but made do with an older version on my
    netbook. And I've never been a professional programmer, the most I've
    ever sold is a few bits of PIC assembler and, a very long time ago, a
    few bits of Delphi, so I'm not going to charge in and start hacking
    Linux code. Waiting is.

    --
    Joe

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From The Wanderer@21:1/5 to Celejar on Thu Jul 18 23:20:01 2024
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
    On 2024-07-18 at 13:50, Celejar wrote:

    Andrew M.A. Cater wrote:

    This is not the place for debugging Sid, I'm afraid, there are too
    few

    It's not? Where, then, is the place for debugging Sid?

    I'm no longer anything *close* to an expert in this area (having not run
    sid myself in well over a decade), but my understanding is: debugging
    sid is done in the BTS, collaboratively among Debian developers and that
    subset of non-DD users who run sid, and preferably involves those users investigating causes and filing bug reports with the results of those investigations.

    By taking on yourself the risk and burden of running sid, you are
    volunteering to be one of those who helps notice issues before they
    reach testing, and report those issues so that the machinery of the
    archive can stop the package versions which those issues from migrating
    to testing. Ideally, you are also volunteering to be one of those who
    helps the package maintainers track down and fix the issues.

    -- And, by doing that, you are volunteering to help protect those who do
    *not* run sid from having to encounter those issues. It's an admirable
    thing to do, really, if you have the time and energy and so forth to
    spare for it.

    (I just doubt whether a lot of those who currently run sid understand
    that that is what they are volunteering for, given the pattern of "what
    to update against" recommendations that I remember seeing, which was
    rather at odds with what I expected and what I myself would have
    recommended.)

    --
    The Wanderer

    The reasonable man adapts himself to the world; the unreasonable one
    persists in trying to adapt the world to himself. Therefore all
    progress depends on the unreasonable man. -- George Bernard Shaw


    -----BEGIN PGP SIGNATURE-----

    iQIzBAEBCgAdFiEEJCOqsZEc2qVC44pUBKk1jTQoMmsFAmaZhlMACgkQBKk1jTQo MmuYKg/+JBBavzpZU/FPPqkOPhbwNEzGHInmCAeIjAM3lgHq8Sfl3GzFqqscBzuO bOj1KDclKH34ugkecPOW090b7FATg6nCqvwXikWifOnfeRJpaSCXji1tMsIHXxMb 36V7Jxg79At93utZf9r8XK+BTqUqpLxkDsRbmemegdfRC0hvRaYI5ojZA2L6Oa/x 6C+ZrAcOjg7MBXJKlwnbLBeDrRr81hkgRylBCnAnXCuOruLjSbccEmkj/WXjRbez mhs8EPwSNfx2jLswpHMKSb7lX3M03zVbO0CAwaRcF5ATnfQ4c1we/RvUsEDjwkFZ hvhjklTL6hLCPGDXIcSyf1W7TFZbm5SSYiFb4nt4r8ljOMKioWjZoMDSsyn14lP2 B2X7c8gaCn5JArwLO5pHr5n7ehg1rMs8xPtQg/W5ODhMrQDMt7kySplnSpFGCxLP +ar1hdZtkkwHnXLVTYkuiFL9FtYo16ca0oM0iag0XJ/NLmoMLry0dA22KsvEEVqy G4fK6qOFIHvsyLNzaIzwkUrARSyykowbpmKYKv0AqnmefnYC3tPAD/oOifkXSeqY b84g7Njmyh9759x2U7fowM07gbAeLEZinkZnwwNBrstlzFbtJAwxFtG4TwWgtju9 D99lTUshNgA+C0WfiM2ImjsF6bwf
  • From Michael =?utf-8?B?S2rDtnJsaW5n?=@21:1/5 to All on Thu Jul 18 23:30:01 2024
    On 18 Jul 2024 13:47 -0400, from celejar@gmail.com (Celejar):
    I don't mean this to be snarky, but that desire seems incompatible
    with running Debian sid. I honestly think it's an unreasonable
    expectation to want official guides for every transitory broken
    state in a development tree.

    That's fair. I think I meant more that I was just going to stick with
    6.9.8 until this gets sorted out, rather than muck around and deviate
    from the default kernel / initrd build settings without official documentation of the process.

    The process for doing that is setting up an apt pin either to force
    the kernel packages to a particular, known good version, or to block installation of a particular, problematic version of the kernel
    packages.

    I agree with what others have already said. If you're running Sid,
    problems are par for the course, and you're expected to be able to
    either help fix those issues (by filing and collaborating about bug
    reports against the affected packages, and maybe contributing patches
    to fix issues that you run across), solve those problems on your own
    system (and ideally submit patches), or wait them out until the
    package maintainers fix them (either based on bug reports filed by
    others, or by noticing the issue themselves). Sometimes more than one
    of those. Yes, maybe it'll work out great with the particular set of
    packages you have installed; all the more power to you in that case.
    Over time that becomes ever more unlikely, though, because at one
    point or another any package is going to be involved in some sort of
    major transition (the switch to a 64-bit time_t, a major libc upgrade,
    a default init system change, or whatever else might happen as time
    goes on).

    --
    Michael Kjörling 🔗 https://michael.kjorling.se “Remember when, on the Internet, nobody cared that you were a dog?”

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ash Joubert@21:1/5 to Celejar on Fri Jul 19 00:50:01 2024
    On 2024-07-19 02:32, Celejar wrote:
    I'm currently on kernel 6.9.8 (amd64 / Sid). Installing 6.9.9 fails due to running out of space on /boot:
    update-initramfs: Generating /boot/initrd.img-6.9.9-amd64
    zstd: error 70 : Write error : cannot write block : No space left on device E: mkinitramfs failure zstd -q -9 -T0 70
    update-initramfs: failed for /boot/initrd.img-6.9.9-amd64 with 1.
    It turns out that the initrd for 6.9.9 is more than 7x the size of the
    one for 6.9.8!

    I had the same problem. The cause is not the kernel upgrade but the
    refactoring of non-free firmware packages. firmware-misc-nonfree now
    recommends firmware-nvidia-graphics, causing it to be installed:

    $ apt-cache show firmware-misc-nonfree
    [...]
    Recommends: firmware-nvidia-graphics, firmware-intel-graphics, firmware-intel-misc, firmware-mediatek


    I am not using an nvidia gpu so the fix for me was:

    apt-get purge firmware-nvidia-graphics


    There was NEWS when I dist-upgraded:

    $ zcat /usr/share/doc/firmware-misc-nonfree/NEWS.Debian.gz
    firmware-nonfree (20230625-3~exp1) experimental; urgency=medium

    Several firmware files were moved from firmware-misc-nonfree into
    their own package:
    - firmware-nvidia-graphics: This package now holds the firmware files for
    Nvidia GPU hardware.
    - firmware-intel-graphics: This package now holds the firmware files
    for Intel Graphics Media Driver chips (mostly i915) as found in
    'modern' Intel CPUs with integrated graphics in the Broadwell and
    the various 'Lake' CPU series.
    - firmware-intel-misc: This package now holds the firmware files for
    Intel
    devices and chips which do not belong in one of the other Intel
    firmware
    packages. These devices/chips include for example Omni-Path devices,
    Ethernet/Network chips/devices and QuickAssist Technology crypto
    accelerators.
    - firmware-mediatek: This package now holds the firmware files for
    devices and chips made by MediaTek and Ralink, which is part of
    MediaTek.

    -- Diederik de Haas <didi.debian@cknow.org> Thu, 18 Jan 2024 14:00:00
    +0100


    Kind regards,

    --
    Ash Joubert (they/them) <ash@transient.nz>
    Director / Game Developer
    Transient Software Limited <https://transient.nz/>
    New Zealand

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From The Wanderer@21:1/5 to Celejar on Fri Jul 19 00:40:01 2024
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
    On 2024-07-18 at 10:32, Celejar wrote:

    Hello,

    I'm currently on kernel 6.9.8 (amd64 / Sid). Installing 6.9.9 fails due to running out of space on /boot:

    *****
    update-initramfs: Generating /boot/initrd.img-6.9.9-amd64
    zstd: error 70 : Write error : cannot write block : No space left on device E: mkinitramfs failure zstd -q -9 -T0 70
    update-initramfs: failed for /boot/initrd.img-6.9.9-amd64 with 1.
    *****

    It turns out that the initrd for 6.9.9 is more than 7x the size of the
    one for 6.9.8!

    *****
    ~$ ls -l /boot/initrd.img*
    -rw-r--r-- 1 root root 27491557 Jul 8 13:45 /boot/initrd.img-6.9.8-amd64 -rw-r--r-- 1 root root 205739589 Jul 16 14:29 /boot/initrd.img-6.9.9-amd64 *****

    Diffing the two initrd files suggests that the problem stems from the
    fact that 6.9.9 is including the NVIDIA GPU System Processor (GSP)
    firmware in the initrd:

    https://download.nvidia.com/XFree86/Linux-x86_64/510.39.01/README/gsp.html

    Arch dealt with this 6 months ago - they claim that the problem
    actually began in kernel 6.7:

    https://bbs.archlinux.org/viewtopic.php?id=291900

    From relatively deep in this thread, the issue appears to be that the
    GSP firmware references the same files multiple times via directory
    symlinks, and mkinitcpio was not detecting that, but rather was creating
    an actual new directory and populating it with additional copies of the
    files for each such symlink.

    The (a?) proposed solution appears to have been to update mkinitcpio so
    that it *does* detect this: "if the parent directory of the current file
    to be added is a symlink, create that symlink instead of adding the
    file", or logic to that effect. I haven't followed far enough to see
    whether that was actually done, but it seems(?) to have gotten far
    enough for a patch doing it to have been proposed.

    I imagine that the solution for the Debian side may wind up being
    analogous, albeit probably for mkinitramfs (or some tool it relies on),
    since I don't see a mkinitcpio in at least the parts of the archive I
    can quickly search against.

    --
    The Wanderer

    The reasonable man adapts himself to the world; the unreasonable one
    persists in trying to adapt the world to himself. Therefore all
    progress depends on the unreasonable man. -- George Bernard Shaw


    -----BEGIN PGP SIGNATURE-----

    iQIzBAEBCgAdFiEEJCOqsZEc2qVC44pUBKk1jTQoMmsFAmaZl4wACgkQBKk1jTQo Mmt0OQ/9H03OaLkJysZgdkYjQel3veCZg19yzzWb4BZSIkS5aBiXQYAT6NrKB4MR ZUxPTw8k0vkOkYuWsM+h03QfOrQ3hoypi9ZpzF5loDq70+vwleW+oFYqng70K2fg 9jcvIwAQ8kf5+Hpdk/Sdlqy3EwN6WV+pWdytrdXrHhViDPNM05rGtbTrCr+6sx19 zdt9akBkhFXwGhCLhjDM182aWmnIfF1/qAMmWtshHa4NI4t7CZW8vGgjnTW24gHi iDt6hZCkg1h4/5KGU1/LXRiV1PvW5onbACGZ+7T9b7C2LnLNWOwRLa7KJb8ORTsa ZJSAF97AR1IApRqXtlA0RI7garb98Alo/mk5h2voCjzIzAMvZm6e/89ptZpSfkZK Ov07cp1HdXIhw8YeHkeMnFT2DrKGMp/xjm4NMqhQvEerK0mF4IVcluX7xbvZ9a49 XLQsdLo+wjBzvI5As+2f9kvayTaXtkPqpPHt0dOc466lPD72c/Hhas5+WHgYz17H pK5KsWtUlyR2M02bWKIBflGDT/OIJ2xs8EGWgbXjJypEwGMV1eZXKEOlJIyjnQkU rEIWvWpA4FCR1RIaaLKCvZ6x/NV8CWUaGDsXPd7g2r7Rsbv/vSUORsgSAKtyMb2q De8oSFwc33OnsDGc4FBP60GaYzHd
  • From songbird@21:1/5 to The Wanderer on Fri Jul 19 05:40:01 2024
    The Wanderer wrote:
    ...
    By taking on yourself the risk and burden of running sid, you are volunteering to be one of those who helps notice issues before they
    reach testing, and report those issues so that the machinery of the
    archive can stop the package versions which those issues from migrating
    to testing. Ideally, you are also volunteering to be one of those who
    helps the package maintainers track down and fix the issues.

    there is no such contract. just like Debian Developers and
    other volunteers are not forced to do work they would not wish
    to do. however, Debian Developers and others do have a social
    agreement and other rules they do pledge to abide by.

    a simple user? anyone using any version, there's no contract
    we agree to for such use.


    -- And, by doing that, you are volunteering to help protect those who do *not* run sid from having to encounter those issues. It's an admirable
    thing to do, really, if you have the time and energy and so forth to
    spare for it.

    (I just doubt whether a lot of those who currently run sid understand
    that that is what they are volunteering for, given the pattern of "what
    to update against" recommendations that I remember seeing, which was
    rather at odds with what I expected and what I myself would have recommended.)

    i usually run testing and parts of unstable, but also stable.

    i contribute where i can when i can, but i'm certainly not
    bound by any formal agreement or contract (nor could i real-
    istically honor one were it somehow attempted to be imposed).

    i understand your intent and that is how i feel myself but i
    would not state it so strongly. i consider it all on my best
    efforts and under the time contraints i have. sometimes i can
    be interrupted at any moment caring for a parent so that is
    a first priority no matter what.


    songbird

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ash Joubert@21:1/5 to Celejar on Sat Jul 20 02:20:01 2024
    On 2024-07-20 03:39, Celejar wrote:
    Thanks much!
    [...]
    As per another message in this thread, I've already filed a bug against linux-image-6.9.9-amd64, but I suppose I should update the report with
    this information, indicating that it's not really a problem with that package.

    You are welcome! Please close your kernel bug report.

    Hard to pin this on the firmware package either as the recommends is
    arguably the right behaviour to prevent missing firmware at boot time,
    but the recommends has this surprising side-effect for those with a
    /boot partition with insufficient free space for such a large firmware
    package. It is not dangerous because the old initrd is not removed until
    the new one is written successfully, but it is annoying!

    Kind regards,

    --
    Ash Joubert (they/them) <ash@transient.nz>
    Director / Game Developer
    Transient Software Limited <https://transient.nz/>
    New Zealand

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From David Wright@21:1/5 to Ash Joubert on Sat Jul 20 06:10:02 2024
    On Sat 20 Jul 2024 at 12:13:28 (+1200), Ash Joubert wrote:
    On 2024-07-20 03:39, Celejar wrote:
    Thanks much!
    [...]
    As per another message in this thread, I've already filed a bug against linux-image-6.9.9-amd64, but I suppose I should update the report with
    this information, indicating that it's not really a problem with that package.

    You are welcome! Please close your kernel bug report.

    Hard to pin this on the firmware package either as the recommends is
    arguably the right behaviour to prevent missing firmware at boot time,
    but the recommends has this surprising side-effect for those with a
    /boot partition with insufficient free space for such a large firmware package. It is not dangerous because the old initrd is not removed
    until the new one is written successfully, but it is annoying!

    According to Debian policy: "The Recommends field should
    list packages that would be found together with this one
    in all but unusual installations."
    -----------------------------

    So if my old Broadcom ethernet card requires firmware-misc-nonfree
    (the package for "firmware blobs which are not individually large
    enough to warrant a standalone package"), it would be /unusual/ for
    the installation not to need some Nvidia firmware as well?

    That doesn't seem to make any sense. Suggests might be a better
    relationship, but even that seems a stretch to me.

    "Suggests […] is used to declare that one package may be more useful
    with one or more others. Using this field tells the packaging system
    and the user that the listed packages are related to this one and
    can perhaps enhance its usefulness, but that installing this one
    without them is perfectly reasonable."

    Cheers,
    David.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ash Joubert@21:1/5 to Celejar on Wed Jul 31 06:50:01 2024
    On 2024-07-31 16:00, Celejar wrote:
    Update: FWIW, Debian Developer Ben Hutchings actually assigned this
    issue a "grave" severity, and it was ultimately moved to the
    initramfs-tools package. It's now fixed: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1076561

    Well done! Thank you for helping to make Debian better. 🙏

    Kind regards,

    --
    Ash Joubert (they/them) <ash@transient.nz>
    Director / Game Developer
    Transient Software Limited <https://transient.nz/>
    New Zealand

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)