Hi,
GNU Make now has a --shuffle option that simulates non-deterministic
ordering of target prerequisites. See https://trofi.github.io/posts/238-new-make-shuffle-mode.html and also previous work in Debian by Santiago Vila: https://people.debian.org/~sanvila/make-shuffle/
While make always processes prerequisites left-to-right when running sequentially, prerequisites can be evaluated in an arbitrary order when building in parallel (make -j). This option is thus useful to trigger
and debug issues that occur when building in parallel.
I identified 511 packages in unstable[0] (dd-list[1]) that fail to build
with 'make --shuffle=reverse' or 'make --shuffle=random', but do not fail to build with 'make --shuffle=none'.
[0] http://qa-logs.debian.net/2025/05/05/shuffle/sources.txt
[1] http://qa-logs.debian.net/2025/05/05/shuffle/dd-list.txt
Builds logs are available in http://qa-logs.debian.net/2025/05/05/shuffle/
I would like to submit bugs (severity:minor) against those packages.
I'd rather do the bug submitting now to avoid refreshing build results
later, but I agree that this is not release-critical material of course,
so I can also wait until after the trixie release.
One could argue that those issues are not bugs in Debian since they
cannot be triggered in packages that do not build in parallel. However building in parallel is now the default, so I think that even for those packages that do not build in parallel yet, it is good to know that
there is something broken that could bite with a difficult-to-reproduce-and-explain race condition if parallel building
was enabled.
[...]
I'd rather do the bug submitting now to avoid refreshing build results
later, but I agree that this is not release-critical material of course,
so I can also wait until after the trixie release.
El 5/5/25 a las 21:26, Lucas Nussbaum escribi:
[...]
Thanks a lot for this. I was never brave enough to go ahead
and announce a MBF.
May I know what kind of machines did you use to found those bugs?
Machines with 8 CPUs only? (I ask because I found more than 800
packages with makefile issues triggered by make --shuffle,
using machines with 1 and 2 CPUs).
I'd rather do the bug submitting now to avoid refreshing build results later, but I agree that this is not release-critical material of course,
so I can also wait until after the trixie release.
Debian Policy does not say "Makefiles should be correct", but I think
that has always been implicit. The GPL speaks about Makefiles as an integral part of source code ("plus the scripts used to control compilation
and installation of the executable" in GPL-2).
So, I would say this is a normal bug.
In some cases, the bug is already known, because debian/rules
has --max-parallel=1. Example: The alpine package.
(I wonder how much feasible would be to skip those packages)
On 05/05/25 at 22:14 +0200, Santiago Vila wrote:
In some cases, the bug is already known, because debian/rules
has --max-parallel=1. Example: The alpine package.
(I wonder how much feasible would be to skip those packages)
The alpine package is indeed a good example of a package that makes
extensive use of the sequentiality of 'make', and that is going to be
hard to adjust to switch to parallel building or arbitrary orders.
However I still think that there's value in filing bugs for such
packages, because --shuffle=reverse makes it much easier to debug such issues: instead of trying a parallel build and getting a subtlely
different race conditions at each run, you get a reproducible ordering
that exhibits one issue that you can debug, and then move on to the next issue.
Also it's not trivial to distinguish between packages that do not build
in parallel on purpose, vs those that just happen not to build in
parallel (yet).
Lucas
What is a maintainer supposed to do when the package already does
"dh --no-parallel" and the upstream Makefiles are basically unfixable?
How many of the packages that break with "make --shuffle" are currently
doing parallel building?
I am asking since these might be RC bugs for trixie.
El 14/5/25 a las 12:50, Adrian Bunk escribió:
...
How many of the packages that break with "make --shuffle" are currently doing parallel building?
I am asking since these might be RC bugs for trixie.
I believed (regarding build failures) that you only cared about RC-ness
when the failure happens in the buildds. Have you changed your mind?
Thanks.
On Wed, May 14, 2025 at 06:58:22PM +0200, Santiago Vila wrote:
El 14/5/25 a las 12:50, Adrian Bunk escribió:
...
How many of the packages that break with "make --shuffle" are currently
doing parallel building?
I am asking since these might be RC bugs for trixie.
I believed (regarding build failures) that you only cared about RC-ness
when the failure happens in the buildds. Have you changed your mind?
Exotic setups like single-cpu or swap-starved are not something that
could happen on release architecture buildds, I do consider it harmful
trying to force maintainers to waste time on supporting unsuitable build environments through RC bugs.
We will support the needs of our users for operation in many
different kinds of computing environments.
Single-CPU systems are ubiquitous in the cloud. I was a system admin in
a small startup some time ago. Everything was in the cloud, and one of my duties was naturally to reduce the IT bill if possible, so I used single-cpu systems extensively, as they are almost always cheaper.
Filing these bug reports sounds like a good idea to me. I don’t see
any reason to wait as these will be severity:minor, so they won’t
interfere with the trixie release.
El 17/5/25 a las 1:13, Cyril Brulebois escribió:
Soren Stoutner <soren@debian.org> (2025-05-05):
Filing these bug reports sounds like a good idea to me. I don’t see any reason to wait as these will be severity:minor, so they won’t interfere with the trixie release.
Filing now can trigger uploads to fix those minor bugs, meaning packages that absolutely do not meet requirements for freeze exceptions end up in unstable. Depending on circumstances, they might make migrations of other packages more complicated than they need to be.
(I'll refrain from naming a particular example, but I'm aware of one of those without even searching for it, it just happened to show up on one of my specific “important for the release” radars…)
Hi. Discussing about the right time to report those bugs does not make
much sense anymore, because Lucas already reported them :-)
Now we can only expect maintainers to act responsibly, and as you
point out, be particularly careful about not affecting other packages.
I think the idea of reporting them now was more about letting the bugs
to be known "soon", more than starting to fix them "soon". Now we can
forward the bugs upstream and some of the fixes will be present in the
new upstream releases that we will start to upload after the release.
Soren Stoutner <soren@debian.org> (2025-05-05):
Filing these bug reports sounds like a good idea to me. I don’t see
any reason to wait as these will be severity:minor, so they won’t
interfere with the trixie release.
Filing now can trigger uploads to fix those minor bugs, meaning packages
that absolutely do not meet requirements for freeze exceptions end up in unstable. Depending on circumstances, they might make migrations of other packages more complicated than they need to be.
(I'll refrain from naming a particular example, but I'm aware of one of
those without even searching for it, it just happened to show up on one of
my specific “important for the release” radars…)
Santiago Vila <sanvila@debian.org> (2025-05-17):
El 17/5/25 a las 1:13, Cyril Brulebois escribió:
Soren Stoutner <soren@debian.org> (2025-05-05):
Filing these bug reports sounds like a good idea to me. I don’t see any reason to wait as these will be severity:minor, so they won’t interfere with the trixie release.
Filing now can trigger uploads to fix those minor bugs, meaning packages that absolutely do not meet requirements for freeze exceptions end up in unstable. Depending on circumstances, they might make migrations of other packages more complicated than they need to be.
(I'll refrain from naming a particular example, but I'm aware of one of those without even searching for it, it just happened to show up on one of
my specific “important for the release” radars…)
Hi. Discussing about the right time to report those bugs does not make
much sense anymore, because Lucas already reported them :-)
Given I mentioned spotting a resulting upload, it should be pretty clear
that I do know that has happened.
It seemed important to me to reply to the “I don't see any reason to wait” part (which I quoted, and kept above just in case), so that people might take that in consideration next time they want to MBF that late in
the freeze.
Now we can only expect maintainers to act responsibly, and as you
point out, be particularly careful about not affecting other packages.
The example I was alluding to builds a udeb, and is therefore very much
not something like a leaf package one can easily pretend doesn't exist.
I think the idea of reporting them now was more about letting the bugs
to be known "soon", more than starting to fix them "soon". Now we can forward the bugs upstream and some of the fixes will be present in the
new upstream releases that we will start to upload after the release.
I didn't challenge any reasons *for* filing. I'm giving a reason *not
to*, in reply to someone who said wasn't seeing any.
Also, as a random maintainer, despite the “Severity: minor”, seeing
piles of FTBFS bug reports pop up all of a sudden in my maildir right
before entering hard freeze does not bring any kind of warm, cozy
feeling. Distraction and worry isn't quite something I enjoy. YMMV.
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 546 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 149:32:57 |
| Calls: | 10,383 |
| Calls today: | 8 |
| Files: | 14,054 |
| D/L today: |
2 files (1,861K bytes) |
| Messages: | 6,417,769 |