I have been looking into the aliasing problems in dpkg on behalf of Freexian's Debian funding. To that end I proposed a possible way forward
last year (https://lists.debian.org/debian-dpkg/2022/11/msg00007.html),
but the feedback I got was not particularly helpful in determining
consensus.
A little later, Simon Richter also looked into the problem (https://lists.debian.org/debian-dpkg/2022/12/msg00023.html), but
remained silent after the initial post. Little happened since then. Now Raphael Hertzog proposed to use the DEP process to get this thing
unstuck
and with the help of Emilio Pozuelo Monfort I created a draft
for discussion. I allocate number 17 via debian-project@l.d.o. What
follows is the draft text. Please consider it to be a piece of best intentions at reconciling feedback wherever I could.
Introduction
============
At its core, `dpkg` assumes that every filename uniquely refers to a
file on disk. The situation where two distinct filenames refer to the
same file on disk is referred to as aliasing.
Proposal
========
In order to handle aliasing efficiently, `dpkg` gains new options `--add-alias <symlink>`, `--remove-alias <symlink>` and
`--list-aliases`. When creating symbolic links that cause aliasing
effects, the creating entity is supposed to inform `dpkg` using an appropriate invocation. Doing so records the aliasing information in a
new mapping inside its administrative directory. No existing
administrative files are modified as a result of this operation. When
`dpkg` operates on paths, it can compute a canonicalized version using a
pure function without the need to `stat()` files on disk thus greatly improving performance. Canonicalized paths are only needed when
determining whether a file conflict exists. In all other cases,
original paths continue to be used as symbolic links will be followed by filesystem operations. The `--add-alias` operation records the target
of the symbolic link that must exist prior to invocation. The `--remove-alias` operation fails if any files are still installed in the aliased location.
Rejected proposals
==================
Hardcoding aliases into dpkg
----------------------------
It was suggested to include a static aliasing mapping into the `dpkg`
source code. Since `dpkg` is used by multiple projects in different
ways (not necessarily Debian-derivatives), this approach would break
other consumers. Also note that Debian's `dpkg` can be used to operate
on an installation using different aliases via the `--root` flag. As
such the alias mapping needs to be a property of the installation.
Modifying package lists in place
--------------------------------
`dpkg` could rewrite the extracted `.list` files from `control.tar` and
store paths in canonicalized form. Canonicalization would happen as
when a `control.tar` is extracted. It would also happen either as a
one-time conversion during the upgrade of `dpkg` or whenever a `.list`
file is read. Given canonicalized list files, string comparison on
files would support conflict detection. Other pieces to be updated in a similar way include `alternatives`, `diversions`, `statoverride`, and `triggers`.
This would affect the output of `dpkg -S`, which would then output canonicalized paths. Packages generated by `dpkg-repack` would have
their contents canonicalized as well.
Managing the aliasing mapping using a control file --------------------------------------------------
It was suggested that the mapping could be managed via a special control
file `canonical`. Given that aliasing is not a common operation, the
benefit of handling it declaratively is minor. Beyond that, aliasing
can also happen as an customization issued by an administrator.
Therefore, a command line based approach is preferred.
Having dpkg move files and create symbolic links ------------------------------------------------
When instructed with `--add-alias`, `dpkg` could also create the corresponding symbolic links and move the affected files to their new location. While that would be convenient, doing so is non-trivial in an atomic way. Sometimes, the underlying filesystem does not fully conform
to POSIX (e.g. `overlayfs`) and such corner cases need to be managed individually. Since such an implementation already exists outside
`dpkg` and its complexity is non-trivial, the moving of files shall
remain external. In case aliases are setup in a bootstrap setting, no
moves are necessary.
Implement aliasing after metadata tracking ------------------------------------------
The [metadata tracking](https://wiki.debian.org/Teams/Dpkg/Spec/MetadataTracking)
feature enhances `dpkg` with knowledge about filesystem metadata for installed files. This includes knowledge of symbolic links, which would
help with tracking aliasing. Unfortunately, progress on this is fairly
slow and we think that aliasing support is more urgent.
I thought my reply was rather clear, and that we had further clarified
that privately, that at the time I thought there was no other answer
required as (AFAIR) you stated you'd be digging further on it. And I mentioned I'd try to reply to the list, but it didn't feel urgent given
the clarifications given, neither the timing during the freeze?
Sigh, a DEP(!?), for a dpkg change? It feels more like a way to exhort pressure over this than anything else TBH…
I'm unlikely to discuss this topic on debian-devel, given previous
nastiness and abuse.
The text includes most (but not all) of what I've been saying publicly,
and what I've tried to further clarify to you and Emilio in private.
But I think ignores the essence of what I've been repeating all along.
I already mentioned this in my reply for the thread you reference. So,
let me repeat and possibly expand to avoid any future doubt. I already considered and discarded something like this (except for using a config option instead of a new command, but that does not really change the substance of the problems).
Let's also get back to the very basics. dpkg manages objects shipped
in binary packages, on the filesystem. It assumes this managing role in exclusivity, it will for example overwrite unmanaged files. It preserves admin changes with interfaces specifically provided for that (diversions, statoverrides, conffile changes) or the unfortunate symlink redirects.
These shipped objects define the filesystem layout (not the other way around). Due to the missing fsys metadata, where it does not have all
such metadata at hand when necessary (it might only have the one for
the currently unpacked .deb), it might use heuristics or check the
filesystem for such metadata, because it does not have anything else,
but that should not be taken to mean that the filesystem is the source
of truth, as most of those will be unnecessary once it has such
metadata at hand.
So the reason this proposal is still conceptually wrong is manifold:
* dpkg cannot safely and atomically perform such switches (and I don't
see it ever being able to portably do so, so I don't see ever
supporting that).
* No packages ships those symlinks (and none should! as that would
currently imply having the same pathname contain different file types
on the same system, introducing ordering issues and file type
conflicts).
* This introduces a series of commands to let dpkg know that a
filesystem change that was not shipped in any .deb (even though that
should have been the way to do it), has been done, which:
- Switches the source of truth from the .deb to the fsys.
- Confuses admin initiated changes from distro initiated ones.
* Wants to be a generic change but it is really targeted to this
specific mess. We have been doing similar aliasing transitions for
many doc dirs, by stopping shipping files within, shipping that
pathname as a symlink and then switching the directories to symlinks
to match (via the dpkg-maintscript-helper hack because we miss fsys
metadata). This means we'd need to then register all these directories
too? Meh.
* This information can get out of sync with reality, as it adds an
additional and unconnected with anything source of truth, that dpkg
cannot do anything about if it diverges (in contrast to diversions
or statoverrides f.ex.). This can never happen when that information
comes from the real source of truth (the fsys metadata via the .deb).
* This also adds undue complexity, by supporting those as admin aliases.
The admin generated redirecting symlinks are already annoying, I'd rather
not add further to that pile. I don't really want to support admins doing
this (dpkg-divert does not even support diverting a directory).
[ As an aside, I think ideally eventually nothing distro provided should
be allowed to be installed within an aliased dir, and dpkg should
eventually just error out in those cases, which eventually would get
rid of the aliasing problems and any such complexity (I'm not sure how
or when that would be feasible though, but obviously in Debian at
least not until nothing ships files there). ]
So this still looks like a terrible interface, like it did at the time
it was discarded; founded on a hack, an interface that seems wants to
be kind of a file-type override but it cannot be, and cannot even
properly act as record tracker, etc…
I thought it would be clear that if there is stuff that depends on
any of this kind of changes to dpkg, relying on those changes in
Debian would not be possible until after trixie+1. Of course there is
always the route to further pile up over the Jenga tower of hacks,
by for example adding huge amounts of Pre-Depends…
So given the above, I don't see why the apparent rush here. And as I've mentioned many times now, I'm planning to continue working on the fsys metadata stuff for 1.22.x, probably at the cost of database duplication
if necessary, if current blockers have not adapted by then. But as I've mentioned before, that might not guarantee this support is sufficient to support fixing this mess. But all other proposed changes I've seen
flying around for changes to dpkg are just conceptually wrong in one way
or another.
Yes, I am quite busy, but it's not forgotten. I keep adding new test cases.
Dpkg already has defined behaviour for directory vs symlink: the directory wins. In principle a future version of dpkg could change that, but /lib/ld-linux.so.2 is just too special, we'd never want to have a package that actually moves it.
That's why I went with "this needs to be a separate mechanism."
The reason to use a control file instead of a tool would be to install the alias from an Essential package, so the old-school "unpack essential packages, then overwrite with dpkg" approach to system installation would work again without special-casing usrmerge in debootstrap&co.
It was suggested that the mapping could be managed via a special control file `canonical`. Given that aliasing is not a common operation, the benefit of handling it declaratively is minor. Beyond that, aliasing
can also happen as an customization issued by an administrator. Therefore, a command line based approach is preferred.
The advantage is that it works for Essential packages, like the one shipping /lib/ld-linux.so.2.
On Sat, Apr 08, 2023 at 04:35:25AM +0200, Guillem Jover wrote:
Let's also get back to the very basics. dpkg manages objects shipped
in binary packages, on the filesystem. It assumes this managing role in exclusivity, it will for example overwrite unmanaged files. It preserves admin changes with interfaces specifically provided for that (diversions, statoverrides, conffile changes) or the unfortunate symlink redirects. These shipped objects define the filesystem layout (not the other way around). Due to the missing fsys metadata, where it does not have all
such metadata at hand when necessary (it might only have the one for
the currently unpacked .deb), it might use heuristics or check the filesystem for such metadata, because it does not have anything else,
but that should not be taken to mean that the filesystem is the source
of truth, as most of those will be unnecessary once it has such
metadata at hand.
This captures an insight I previously didn't have in that clarity and
that I find agreeable conceptually.
So the reason this proposal is still conceptually wrong is manifold:
* dpkg cannot safely and atomically perform such switches (and I don't
see it ever being able to portably do so, so I don't see ever
supporting that).
I agree, but the proposal also does not ask dpkg to perform such
switches, so I kinda fail to see how this is a relevant argument.
* No packages ships those symlinks (and none should! as that would
currently imply having the same pathname contain different file types
on the same system, introducing ordering issues and file type
conflicts).
I disagree with this argument on two levels. For one thing, I think that
the transition only is complete once these symlinks are shipped in a
package. In particular, that notion of complete likely encompasses that
no aliasing occurs anymore as all aliased files have been moved to their canonical location somehow (<- and this likely will be a quite difficult thing to do). For another, no package actually ships those symlinks now.
They are created behind dpkg's back in some postinst. This is
unfortunate and I agree with Simon Richter that this kinda is a policy violation, but at this time, it is an aspect we have to deal with
whether we want to or not.
I suspect that you disagree with the notion the we have to deal with
this situation, which I consider to be our fundamental disagreement.
* This introduces a series of commands to let dpkg know that a
filesystem change that was not shipped in any .deb (even though that
should have been the way to do it), has been done, which:
- Switches the source of truth from the .deb to the fsys.
While this is correct on some level, the aim of this change is to put
that truth back into dpkg of course.
- Confuses admin initiated changes from distro initiated ones.
I think we already do this with dpkg-divert, dpkg-statoverride and other tools. While this may not be nice, it certain has prior art and is
consistent with how we have been doing things in the past.
* Wants to be a generic change but it is really targeted to this
specific mess. We have been doing similar aliasing transitions for
many doc dirs, by stopping shipping files within, shipping that
pathname as a symlink and then switching the directories to symlinks
to match (via the dpkg-maintscript-helper hack because we miss fsys
metadata). This means we'd need to then register all these directories
too? Meh.
I would love to agree with this, but I believe that this ship has
sailed. This likely is part of our fundamental disagreement.
* This information can get out of sync with reality, as it adds an
additional and unconnected with anything source of truth, that dpkg
cannot do anything about if it diverges (in contrast to diversions
or statoverrides f.ex.). This can never happen when that information
comes from the real source of truth (the fsys metadata via the .deb).
I have difficulties accurately capturing the argument. The problem of information getting out of sync with reality should affect every aspect
of dpkg and indeed, that kinda is the status quo where upgrades can
loose files, because dpkg has an incomplete picture of reality. The aim
of this change is to allow us to re-sync the status quo into dpkg. My
view is that dpkg's information presently is out of sync with reality
and the proposed change partially fixes that.
[ As an aside, I think ideally eventually nothing distro provided should
be allowed to be installed within an aliased dir, and dpkg should
eventually just error out in those cases, which eventually would get
rid of the aliasing problems and any such complexity (I'm not sure how
or when that would be feasible though, but obviously in Debian at
least not until nothing ships files there). ]
It seems to me that this is something everyone agrees on. So our
disagreement resides in the way to get there rather than where to get
to.
So this still looks like a terrible interface, like it did at the time
it was discarded; founded on a hack, an interface that seems wants to
be kind of a file-type override but it cannot be, and cannot even
properly act as record tracker, etc…
I agree that in a perfect world, we would not need this. Let me circle
back to our fundamental disagreement.
My impression is that at this time basically everyone except you agrees
that we have to deal with the aliasing problems that have been rolled
out to users and will be forced in bookworm. I believe that this is the
state that we have to consider as starting point and that we cannot
magically turn this transition back to perform it in a better way. And indeed, I believe that there would have been a better way[1] that no
longer is available to us.
On the other hand, my impression is that you continue to see the
transition as fundamentally broken and in a state that we cannot work
from. You appear to believe that if we want to do it, we must start over
in a better way. That better way must not cause aliasing problems to
dpkg.
I thought it would be clear that if there is stuff that depends on
any of this kind of changes to dpkg, relying on those changes in
Debian would not be possible until after trixie+1. Of course there is always the route to further pile up over the Jenga tower of hacks,
by for example adding huge amounts of Pre-Depends…
I agree that we probably will deal with this until at least trixie+1.
This is precisely why I would like to have a plan to finish it sooner
rather than later.
I don't think we disagree (?), I probably didn't express myself clearly.
The fact that no package ships those symlinks *is* and *has* been a
problem, and what I've been saying all along, this will be the only
correct way to let dpkg know whether there will be aliasing in play.
But given these mentioned constraints
it cannot be made to support (as in accept) unpacking files inside
aliased directories (it should be able to unpack the symlinks creating
those aliased directories though!).
dpkg-divert distinguishes between local and package level changes, it
is true that dpkg-statoverride does not have (currently) that
distinction, although it is primarily an admin tool where I don't
think it makes much sense to support something like declarative
package statoverrides TBH once we can ship fsys metadata (perhaps
conditional one though).
On 6/21/23 20:33, Guillem Jover wrote:
I don't think we disagree (?), I probably didn't express myself clearly. The fact that no package ships those symlinks *is* and *has* been a problem, and what I've been saying all along, this will be the only
correct way to let dpkg know whether there will be aliasing in play.
I've looked into building a dpkg-alias tool that would work similar to dpkg-divert, and currently that looks like it might be a viable solution.
The package would need to unregister on upgrade in the postrm though, but that is standard for removed diversions.
- dpkg-query returns the package name if any aliased name matches
There should also be a flag whether to report the file name from the
data.tar as well, defaulting to "no", because that's what scripts expect.
But given these mentioned constraints
it cannot be made to support (as in accept) unpacking files inside
aliased directories (it should be able to unpack the symlinks creating those aliased directories though!).
I think that can be done. I have already successfully made it report a conflict between /bin/testfile and /usr/bin/testfile, with a meaningful
error message, and runtime overhead isn't too bad -- a factor of
log_{262144} 2 on the lookup time for a single path, but inserts got a bit more expensive because these now have prefix comparisons on the path. The latter could probably be improved with another hash on the first N bytes of the path.
I'd like to see a mechanism that ensures that dpkg understands those control files, though -- like a "critical" flag.
I suspect that for trixie, this will have to be an archive side check that any package using one of the declarative interfaces depends on an
appropriate version of dpkg, and/or its use disallowed until trixie+1 for
the convenience of backporters.
Please consider it to be a piece of best
intentions at reconciling feedback wherever I could. At the time of this writing it certainly is not consensus, but consensus is what I seek
here. Without further ado, the full DEP text follows after my name
while it also is available at https://salsa.debian.org/dep-team/deps/-/merge_requests/5
From what I have understood, Guillem would rather avoid committingto a new public interface for this specific use-case, i.e. the
Naive solution
==============
In theory, `dpkg` could resolve this automatically. For every file it touches, it could canonicalize the location using the actual filesystem
and check whether any other installed file has the same canonicalized location. Unfortunately, `dpkg` cannot know which filenames can
collide, so it would check every filename in its database. For canonicalization, it would `stat()` every component of every filename.
This easily amounts to a million or more `stat()` calls on larger installations. Caching could reduce the impact somewhat, but since
Debian introduces aliases during maintainer scripts, it would have to invalidate the cache after maintainer scripts have been run. The
resulting performance would be unacceptable.
Implement aliasing after metadata tracking ------------------------------------------
The [metadata tracking](https://wiki.debian.org/Teams/Dpkg/Spec/MetadataTracking)
feature enhances `dpkg` with knowledge about filesystem metadata for installed files. This includes knowledge of symbolic links, which would
help with tracking aliasing. Unfortunately, progress on this is fairly
slow and we think that aliasing support is more urgent.
Hello,
On Mon, 03 Apr 2023, Helmut Grohne wrote:
Please consider it to be a piece of best
intentions at reconciling feedback wherever I could. At the time of this writing it certainly is not consensus, but consensus is what I seek
here. Without further ado, the full DEP text follows after my name
while it also is available at https://salsa.debian.org/dep-team/deps/-/merge_requests/5
I'd like to express some disappointment that nobody replied publicly
sofar. Last year's developer survey concluded that "Debian should complete the merged-/usr transition" was the most important project for Debian [1] (among those proposed in the survey). That's what we are trying to do
here and it would be nice to build some sort of consensus on what it means
in terms of changes for dpkg.
I know that Guillem (dpkg's maintainer) is generally opposed to the
approach that Debian has followed to implement merged-/usr but I have
yet to read his concerns on the changes proposed here
Here you are considering all files, but for the purpose of our issue,
we can restrict ourselves to the directories known by dpkg. We really
only care about directories that have been turned into symlinks (or
packaged symlinks that are pointing to directories). That's a a much lower number of paths that we would have to check.
Thus this time-consuming operation would be done once, the first
time that the updated dpkg starts and when /var/lib/dpkg/aliases
does not yet exist.
In any case, now that you have a database of aliases, you can do the other modifications to detect conflicting files and avoid file losses.
How does that sound?
The proposal I made above is not a real database in the sense that we
don't record what was shipped by the .deb when we installed the files...
it's rather the opposite, it analyzes the system to detect possible
conflicts with dpkg's view of the system.
It can be seen as complimentary to it. In any case, I don't see how implementing metadata tracking would help to solve the problem that we
have today. dpkg would know that all .deb have /bin as a directory and
not as a symlink, and it would be able to conclude that the directory
has been replaced by a symlink by something external, but that's it.
It should still accept that replacement and do its best to work with it.
After Bookworm ships I plan to propose a policy change to the CTTE and
policy maintainers to forbid shipping files in the legacy directories altogether, followed by a debhelper change to adjust any stragglers automatically at build time and a mass rebuild
Here you are considering all files, but for the purpose of our issue,
we can restrict ourselves to the directories known by dpkg. We really
only care about directories that have been turned into symlinks (or
packaged symlinks that are pointing to directories). That's a a much lower number of paths that we would have to check.
We don't add any new public interface to dpkg, but we also have the possibility to remove to /var/lib/dpkg/aliases to force an new scan
(some sort of "dpkg --refresh-aliases" without an official name).
It might still be cleaner to have that "dpkg --refresh-aliases" command
so that we can invoke it for example in "dpkg-maintscript-helper symlink_to_dir/dir_to_symlink" when we are voluntarily turning a directory into a symlink (or vice-versa).
In any case, now that you have a database of aliases, you can do the other modifications to detect conflicting files and avoid file losses.
How does that sound?
The proposal I made above is not a real database in the sense that we
don't record what was shipped by the .deb when we installed the files...
it's rather the opposite, it analyzes the system to detect possible
conflicts with dpkg's view of the system.
It can be seen as complimentary to it. In any case, I don't see how implementing metadata tracking would help to solve the problem that we
have today. dpkg would know that all .deb have /bin as a directory and
not as a symlink, and it would be able to conclude that the directory
has been replaced by a symlink by something external, but that's it.
After Bookworm ships I plan to propose a policy change to the CTTE and
policy maintainers to forbid shipping files in the legacy directories altogether, followed by a debhelper change to adjust any stragglers automatically at build time and a mass rebuild, plus MBF for the small
% that does not use dh and a piuparts test to stop migration for
anything that is uploaded and doesn't comply. That should bring the
matter to an end, without needing to modify dpkg.
Hi Luca,
On Fri, Apr 21, 2023 at 03:29:33PM +0100, Luca Boccassi wrote:
After Bookworm ships I plan to propose a policy change to the CTTE and policy maintainers to forbid shipping files in the legacy directories altogether, followed by a debhelper change to adjust any stragglers automatically at build time and a mass rebuild, plus MBF for the small
% that does not use dh and a piuparts test to stop migration for
anything that is uploaded and doesn't comply. That should bring the
matter to an end, without needing to modify dpkg.
I agree with the goal of removing aliases by moving files to their
canonical locations. However, I do not quite see us getting there in the
way you see it, but maybe I am missing something. As long as dpkg does
not understand the effects of aliasing, we cannot safely move those
files and thus the file move moratorium will have to be kept in place.
And while moving the files would bring the matter to an end, we cannot
do so without either modifying dpkg or rolling back the transition and starting over. I hope that we all agree that rolling back would be too
insane to even consider, but I fail to see how you safely move files
without dpkg being changed. Can you elaborate on that aspect?
I'd also be interested on how you plan to move important files in
essential packages. This is an aspect raised by Simon Richter and where
I do not see an obvious answer yet.
On Fri, 21 Apr 2023 at 15:29:33 +0100, Luca Boccassi wrote:
After Bookworm ships I plan to propose a policy change to the CTTE and policy maintainers to forbid shipping files in the legacy directories altogether, followed by a debhelper change to adjust any stragglers automatically at build time and a mass rebuild
That seems quite likely to trigger the scenario Helmut is trying to avoid, which if I understand correctly is this:
* foo_12.0 in Debian 12 ships /lib/abcd
* bar_13.0 takes over /lib/abcd from foo, but because of either your
proposed change or a manual action by the maintainer, it is actually in
the data.tar as ./usr/lib/abcd (not ./lib/abcd like it was in foo_12.0)
* the maintainer of bar didn't add the correct Breaks/Replaces on foo
* a user upgrading from Debian 12 to 13 installs bar_13.0, perhaps pulled
in as a dependency
* expected result: dpkg refuses to unpack bar ("trying to overwrite ..."),
the upgrade is cancelled, and the user reports a RC bug in bar
* actual result: /usr/lib/abcd in bar quietly overwrites /lib/abcd from foo
* if bar is subsequently removed, then dpkg (and therefore apt) thinks foo
is fully functional, but in fact /{usr/,}lib/abcd is missing
(For simplicity I've described that scenario in terms of files directly shipped in the data.tar, but dpkg also tracks the ownership of files
created by dpkg-divert or alternatives, and similar things can happen
to those.)
I had hoped that the last section of technical committee resolution
#994388 (which concerns this situation) would become irrelevant in Debian
13, but it's looking as though without the sort of dpkg changes discussed
in this thread, the concern about files moving between packages would
remain a valid concern.
However, as far as I can see, the other reasons not to do this that were mentioned in the last section of #994388 *do* become irrelevant in Debian
13, so solving the files-moved-between-packages thing is the last major blocker for doing what you propose. (Unless someone has a reason why this
is not the case?)
You might reasonably say that "the maintainer of bar didn't add the
correct Breaks/Replaces on foo" is a RC bug in bar - and it is! - but
judging by the number of "missing Breaks/Replaces" bug reports that have
to be opened by unstable users (sometimes me), it's a very easy mistake
to make.
One thing that's particularly tricky about this is that the move from
/ into /usr and the move from foo to bar might be 18 months apart if
they happen to occur at opposite ends of our stable release cycle. In particular, if the move from / into /usr is done as soon as the Debian 13 cycle opens, we cannot predict whether the packages that have undergone
that move will also need to undergo a package split/merge at some point
in the following 18 months (but it's reasonable to assume that at least
some of them will).
Guillem also
raised that this is changing the source of truth from the dpkg database
to the actual filesystem, which Guillem considers wrong and I find that vaguely agreeable.
This is looking at it from a performance point of view. Guillem also
raised that this is changing the source of truth from the dpkg database
to the actual filesystem, which Guillem considers wrong and I find that vaguely agreeable.
We don't add any new public interface to dpkg, but we also have the possibility to remove to /var/lib/dpkg/aliases to force an new scan
(some sort of "dpkg --refresh-aliases" without an official name).
Can I rephrase this as your cache invalidation strategy is that any
external entity (such as a maintainer script) introducing aliases should explicitly invalidate the cache.
If you put it this way, it is not that different from the --add-alias/--remove-alias proposal. It is a different interface to
dpkg, but the semantics are roughly the same:
In both cases, something external to dpkg is responsible for performing
the moves and creating the symbolic links followed by informing dpkg
about the alias (explicitly or implicitly via scanning directories).
Would you agree with me that this is a minor adaption of DEP17? In
The proposal I made above is not a real database in the sense that we
don't record what was shipped by the .deb when we installed the files... it's rather the opposite, it analyzes the system to detect possible conflicts with dpkg's view of the system.
I think that Guillem considers this a bad property as he has expressed
in his reply on debian-dpkg, that .debs should be the source of truth.
Hi,
On Tue, Apr 25, 2023 at 09:07:28PM +0200, Helmut Grohne wrote:
This and /bin/sh is the kind of files I'd consider important. And then
upon thinking further it became more and more difficult for me to make sense of the objection. On a merged system, we can just move that file
to its canonical location without having any trouble even with an unmodified dpkg. So from my pov, the question about important files can
be disregarded. I hope Simon Richter agrees.
Yes, the relevant code at
https://github.com/guillemj/dpkg/blob/main/src/main/unpack.c#L749
already handles moving a file inside the same package, and that has
existed for some time, that's why I use two packages for the PoC.
I have not looked for more issues beyond that, so there might be others lurking in the depths of this code.
What I'm mostly concerned about (read: have not verified either way)
with /lib/ld.so and /bin/sh is what happens when dpkg learns of /bin and
/lib as symlinks -- because right now, the symlinks created by usrmerge
are protected by the rule that if dpkg expects a directory and finds a symlink, that is fine because that is obviously an action taken by the
admin.
But if dpkg sees a package containing these as symlinks, then this is
entered into the dpkg database, and subject to conflict resolution, and
for that, a separate rule exists that directory-symlink conflicts are resolved in favour of the directory, so the interaction between a newer base-files packages shipping /lib as a symlink and an older or
third-party package containing /lib as a directory (e.g. a kernel
package from a hosting provider) could overwrite the /lib symlink.
It might be possible to avoid that by never shipping /lib as a symlink
and always creating it externally, but I still think that's kind of
wobbly.
This and /bin/sh is the kind of files I'd consider important. And then
upon thinking further it became more and more difficult for me to make
sense of the objection. On a merged system, we can just move that file
to its canonical location without having any trouble even with an
unmodified dpkg. So from my pov, the question about important files can
be disregarded. I hope Simon Richter agrees.
If we look deeper into the dpkg toolbox, we pause at diversions. What if
the new package were to add a (--no-rename) diversion for files that are
at risk of being accidentally deleted in newpkg.preinst and then remove
that diversion in newpkg.postinst? Any such diversion will cause package removal of the oldpkg to skip removal of the diverted file (and instead deleted the non-existent path that we diverted to). Thus we retain the
files we were interested in.
"Simon" == Simon McVittie <smcv@debian.org> writes:
Brilliant! Would never have thought of using divert like that.
So, what work would need to happen to make this reality? Do we need tooling/scripts/build changes to support the divert scheme, or is it
"simply" a matter of documenting and testing?
In sincerely hope that this fixed-up plan doesn't have any serious
issues. If you find any please tell.
"Simon" == Simon McVittie <smcv@debian.org> writes:
Simon> You might reasonably say that "the maintainer of bar didn't
Simon> add the correct Breaks/Replaces on foo" is a RC bug in bar -
Simon> and it is! - but judging by the number of "missing
Simon> Breaks/Replaces" bug reports that have to be opened by
Simon> unstable users (sometimes me), it's a very easy mistake to
Simon> make.
Is adding the correct breaks/replaces enough to solve things?
I could believe adding a versioned conflicts would be sufficient, but it
is not obvious to me that breaks/replaces is enough given that dpkg
doesn't understand aliasing.
My intuition (and I have not worked through this as much as you) is that
any time you can have files moving where both packages are unpacked can create problems.
I think that can happen with breaks/replaces but not without a conflicts (without replaces?)
At some point the question becomes: Do we want that complexity inside
dpkg (aka DEP 17 or some variant of it) or outside of dpkg (i.e. what
we're talking about here). It seems clear at this time, that complexity
is unavoidable.
You might reasonably say that "the maintainer of bar didn't add the
correct Breaks/Replaces on foo" is a RC bug in bar - and it is! - but
judging by the number of "missing Breaks/Replaces" bug reports that have
to be opened by unstable users (sometimes me), it's a very easy mistake
to make.
On Wed, 26 Apr 2023 at 10:11, Simon Richter <sjr@debian.org> wrote:
What I'm mostly concerned about (read: have not verified either way)
with /lib/ld.so and /bin/sh is what happens when dpkg learns of /bin and
/lib as symlinks -- because right now, the symlinks created by usrmerge
are protected by the rule that if dpkg expects a directory and finds a
symlink, that is fine because that is obviously an action taken by the
admin.
But if dpkg sees a package containing these as symlinks, then this is
entered into the dpkg database, and subject to conflict resolution, and
for that, a separate rule exists that directory-symlink conflicts are
resolved in favour of the directory, so the interaction between a newer
base-files packages shipping /lib as a symlink and an older or
third-party package containing /lib as a directory (e.g. a kernel
package from a hosting provider) could overwrite the /lib symlink.
It might be possible to avoid that by never shipping /lib as a symlink
and always creating it externally, but I still think that's kind of
wobbly.
IMHO we should not ship the top-level symlinks in a package. The
reason for that is to allow the use case where /usr is a separate
vendor partition and / is either a luks volume or a tmpfs, and thus
the top-level symlinks are ephemeral and re-created on boot on the
fly. If they were part of a package, that would get awkward to say the
least.
I really would like to move toward the direction of having exclusively
/usr and /etc shipped in data.tar, and everything else created locally
as needed, and that includes files in /.
Which part of config-package-dev causes a conflict here? Is it
something that can be fixed? Given it's declarative, an upload + rdeps >rebuild should be all that's needed, assuming we know what the issue
is and how to fix it. As far as I can remember, it's a build-time
utility and everything it does is embedded in the target package's
maintainer scripts. But it's been a few years since I last used it, so
I might remember wrongly.
On Fri, 28 Apr 2023 at 09:09, Helmut Grohne <helmut@subdivi.de> wrote:
So yeah, with the exception of dash, this looks fairly good. Let me also dive into dash. Unlike the majority of diverters, it diverts in postinst rather than preinst to allow controlling /bin/sh via debconf. A similar technique is in effect by gpr. In any case, this is special, because
dash diverts its own files, so when moving dash's file, its diversions
can be migrated at the same time. It merely means, that we cannot have debhelper just move files (as that would horribly break dash) and
instead have to move files on a package-by-package way. We could also
opt for removing dash's diversion in the default case and there even is
a patch for doing so (#989632) since almost two years. Too bad we didn't apply it. In any case, as long as the file moving is not forced via debhelper, dash should be harmless.
If I understand correctly, by "forced via debhelper" you mean the
proposal of fixing the paths at build time, right? But if not via
that, it means having to fix all of them by hand, which is a lot - is
it possible to fix dash instead? Or else, we could add an opt-out via
one of the usual dh mechanisms, and use it only in dash perhaps?
Transforming existing diversions: yes, if you can find out about them
without looking at dpkg internal files. It may very well be necessary to update the file format on one of these, and if that would cause your
script to create nonsense diversions, that'd be another thing we'd have
to work around while building real aliasing support.
My current mood is "I'd rather focus on a proper solution, not another
hack that needs to be supported by the proper solution."
Anything we build here that is not aliasing support for dpkg, but
another "shortcut" will delay aliasing support for dpkg because it adds
more possible code paths that all need to be tested.
Keep in mind that we're here because someone took a shortcut, after all.
From my point of view the only reason to try and solve this with a pileof hacks is get us to a state that the current dpkg can deal well with
My understanding from following this thread (and others) is that dpkg
has a bug that can easily be triggered by a sysadmin replacing a
directory with a symlink (and some other necessary conditions that don't happen very often), which is explicitly allowed by policy. This bug is
the one that is causing the problem with the approach that was chosen by
the people implementing usrmerge, even though they were aware of this
problem and a different approach that would have taken two release
cycles and would not have triggered this bug was considered and
rejected.
If this is correct, then Luca's approach may fix the problem for
usrmerge, but does not fix the general dpkg bug. (And, IIUC, is going
to take two _more_ release cycles to fix the problems with usrmerge as implemented! Hmm...)
The --add-alias solution that has been suggested in this thread seems
like it would fix the general problem iff policy was changed to require sysadmins to use it if they replaced a directory with a symlink.
I do not understand why the dpkg maintainer has rejected this solution;
it would still be a fix for the general bug after the usrmerge
transition has completed. And it would be at least one order of
magnitude more performant than scanning the filesystem for directory symlinks.
I think we have a misunderstanding here. As far as I understand it, the
core idea of Luca's approach is that we move all files to their
canonical locations and then - when nothing is left in directories such
as /bin or /lib - there is no aliasing anymore, which is why we do not
have to teach dpkg about aliasing and never patch it.
This and /bin/sh is the kind of files I'd consider important. And then
upon thinking further it became more and more difficult for me to make sense of the objection. On a merged system, we can just move that file
to its canonical location without having any trouble even with an unmodified dpkg. So from my pov, the question about important files can
be disregarded. I hope Simon Richter agrees.
Yes, the relevant code at
https://github.com/guillemj/dpkg/blob/main/src/main/unpack.c#L749
already handles moving a file inside the same package, and that has
existed for some time, that's why I use two packages for the PoC.
Ok, let's move on. I've proposed diversions as a cure, but in reality diversions are a problem themselves. Consider that
cryptsetup-nuke-password diverts /lib/cryptsetup/askpass, which is
usually owned by cryptsetup. If cryptsetup were to move that file to
/usr, the diversion would not cover it anymore and the actual content of askpass would depend on the unpack order. That's very bad and none of
what I proposed earlier is going to fix this.
So how do we fix diversions? Let's have a look into the dpkg toolbox
again. I've got an idea. Diversions. What you say? How do you fix
diversions with diversions? Quite obviously, you divert
/usr/bin/dpkg-divert! And whenever dpkg-divert is instructed to add a diversion for a non-canonical path, you forward that call to the real dpkg-divert, but also call it with a canonicalized version such that
both locations are covered. When initially deploying the diversion of /usr/bin/dpkg-divert, we also need to transform existing diversions.
After Bookworm ships I plan to propose a policy change to the CTTE and
policy maintainers to forbid shipping files in the legacy directories altogether, followed by a debhelper change to adjust any stragglers automatically at build time and a mass rebuild, plus MBF for the small
% that does not use dh and a piuparts test to stop migration for
anything that is uploaded and doesn't comply. That should bring the
matter to an end, without needing to modify dpkg.
We don't want to stat all the files in all packages but we could do better: if we are about to remove an old file that is available through a
symlinked directory, we could check the new name of the file and see if
it's available in some package... and if yes just forget the file without removing it.
This file removal is the reason of the moratorium and incuring some extra cost in some specific cases (installation through directory symlinks which
is not the default case, and would not affect us after the migration is complete) seems certainly fair.
This is problems we know about now, but it likely is not an exhaustive
list. This list was mostly guided by Guillem's intuition of what could
break at https://wiki.debian.org/Teams/Dpkg/MergedUsr and I have to say
that his intuition was quite precise thus far. Notably missing in the investigation are statoverrides. However, we should also look for a more generic approach that tries capturing unexpected breakage.
"Helmut" == Helmut Grohne <helmut@subdivi.de> writes:
I think there is a caveat (whose severity I am unsure about): In order
to rely on this (and on DEP 17), we will likely have versioned
Pre-Depends on dpkg. Can we reasonably rule out the case where and old
dpkg is running, unpacking a fixed dpkg, configuring the fixed dpkg and
then unpacking an affected package still running the unfixed dpkg
process?
I think the file loss problem is one sufficient reason to have the moratorium. We didn't need other reasons once we knew this one. Now that
we look into dropping the moratorium, we need to ensure that there are
no reasons anymore and we learned that diversions are affected in a non-trivial way. So even if we were to fix just the file loss problem,
the diversion problems would still be sufficient reason to keep the moratorium unless they were also fixed by the approach. Here you need
both directions a) diverting a non-canonical location would have to
divert a canonical file and b) diverting a canonical location would have
to divert a non-canonical file. This is breaking the initial assumption.
In any case, this train of thought is definitely widening the solution
space. Thank you very much.
I noticed that the number of packages shipping non-canonical files is relatively small. It's less than 2000 binary packages in unstable and
their total size is about 2GB. So I looked into binary-patching them and attach the resulting scripts.
What still applies here is that we can have usr-is-merged divert /usr/bin/dpkg-divert and have it automatically duplicate any possibly
aliased diversion and then the diverter may Pre-Depends: usr-is-merged (>=...) to have its diversions duplicated. Of course, doing so will make usr-is-merged very hard to remove, but we have experience here from multiarch-support.
And of course, we can always draw the diversion card and have
usr-is-merged divert /usr/bin/update-alternatives to have it
canonicalize paths as required to be able to migrate alternatives in a
sane way (from a consumer point of view).
I don't know APT well enough to answer that question but from my point of view it's perfectly acceptable to document in the release notes that you
need to upgrade dpkg first.
Are you sure that we need anything for diversions except some documented policy on how to deal with it?
AFAIK the following sequence performs no filesystem changes and should
be sufficient to move a diversion to its new location (I only consider the case of an upgrade, not of a new installation that should just work "normally" on the new location):
dpkg-divert --package $package --remove /bin/foo --no-rename
dpkg-divert --package $package --add /usr/bin/foo --divert /usr/bin/foo.diverted --no-rename
=...) to have its diversions duplicated. Of course, doing so will make usr-is-merged very hard to remove, but we have experience here from multiarch-support.
The case of update-alternatives is likely more tricky. You already looked into it. That's a place where it will be harder to get things right
without some changes.
- it is not an error to register a diversion for an alias of anHow do you distinguish between aliased diversions and "real" ones?
existing diversion, provided the package and target matches, this is a
no-op
- it is not an error to unregister a diversion for an alias of a path
that has been unregistered previously, that is a no-op as well
For aliasing support in dpkg, that means we need a safe policy of dealing with diversions that conflict through aliasing that isn't "reject with error", because the magic dpkg-divert would always generate conflicts.
From my point of view, the ultimate goal here should be moving all filesto their canonical location and thereby make aliasing effects
then a package containing /bin/foo and a package containing /usr/bin/foo now have a file conflict in dpkg. Not sure if that is a problem, or exactly the
behaviour we want. Probably the latter, which would allow us to define a policy "if aliased paths are diverted, the diversion needs to match", which in turn would allow the conflict checker during alias registration to verify that the aliased diversions are not in conflict.
The diverted dpkg-divert would probably generate extra register/unregister calls as soon as dpkg-divert itself is aliasing aware, but all that does is generate warning messages about existing diversions being added again, or nonexistent diversions being deleted -- these are harmless anyway, because maintainer scripts are supposed to be idempotent, and dpkg-divert supports that by not requiring scripts to check before they register/unregister.
We get to draw this card exactly once, and any package that would need the same diversion would need to conflict with usr-is-merged, which would make
it basically uninstallable.
My proposal would be to put the onus on the client registering the[...]
diversion:
- packages are encouraged to register both diversions
On 2023-05-05 Simon Richter <sjr@debian.org> wrote:
[...]
My proposal would be to put the onus on the client registering the diversion:[...]
- packages are encouraged to register both diversions
Hello,
That seems to be a rather ugly user interface, ("There is dpkg-divert on Debian, but because the usrmerge you need to invoke it twce to be
sure"). Will we need to have a meta-transition years from now trying to
get get rid of the double diversions?
Hi,
On 06.05.23 07:11, Luca Boccassi wrote:
- every package is forcefully canonicalized as soon as trixie is open
for business
You will also need to ship at least
- /lib -> usr/lib (on 32 bit)
- /lib64 -> usr/lib64 (on 64 bit)
as a symlink either in the libc-bin package or any other Essential
package, to fulfill the requirement that unpacked Essential packages are operational.
Hi,
On 06.05.23 21:28, Luca Boccassi wrote:
[shipping usrmerge symlinks in packages]
In the far future I'd like for these details to be owned by image builders/launchers/setup processes rather than a package, but this can
be discussed separately and independently, no need to be tied to this effort.
Ideally I'd like to have this information in a single package rather
than distributed over ten different tools, especially as this is also
release and platform dependent.
If possible, I'd like to go back to the gold standard of
- download Essential: yes packages and their dependencies
- unpack them using dpkg --fsys-tarfile | tar -x
- install over this directory with dpkg --root=... -i *.deb
to get something that works as a container. Right now, that only works
if I remove "init" from the package list, which is fair since a
container doesn't need an init system anyway.
The less an image builder needs to deviate from this approach, the
better for our users.
To have a working system you need several more steps that are
performed by the instantiator/image builder, such as providing working
and populated proc/sys/dev, writable tmp/var, possibly etc. And it
needs to be instantiated with user/password/ssh certs/locale/timezone.
And if it needs to be bootable on baremetal/vm, it needs an ESP. And
then if you have an ESP and want to run in a VM with SB, you'll need self-enrolling certs on first use or ensuring the 3rd party CA is provisioned. And then...
You get the point. Going from a bunch of packages to a running system necessarily has many steps in between, some that are already done and
taken for granted, for example when you say "works as a container" I'm
pretty sure the "container" engine is taking care of at the very least proc/dev/sys for you, and it's just expected to work. bin -> usr/bin,
sbin -> usr/sbin and lib -> usr/lib should get the same treatment: if
they are not there, the invoked engine should prepare them. systemd
and nspawn have been able to do this for a while now.
Not having those hard coded means that the use case of / on a tmpfs
with the rest instantiated on the fly, assembled with the vendor's
/usr and /etc trees, becomes possible, which is neat. And said trees
can pass the checksum/full integrity muster.
But then, you only capture diversions inside Debian's ecosystem
I also caution that we've started from a very simple approach and tried fixing it up to address the problems that we recognized in analyzing it.
My impression is that we are not finished with our discovery here and
won't be for quite some time.
But then, you only capture diversions inside Debian's ecosystem and miss
out on other kinds of diversions such as local diversions. We currently support imposing local diversions on pretty much arbitrary files
including unit files.
Hi,
On Sun, 2023-05-07 at 07:50 +0200, Helmut Grohne wrote:
But then, you only capture diversions inside Debian's ecosystem
It's unreasonable to support stuff outside Debian's ecosystem: even
basic dependency relations do not work for this.
Debian's dependency system requires to explicitly declare Depends/Conflicts/Replaces/Breaks, but for obvious reasons we cannot do
that for packages outside Debian's ecosystem.
The same is true for diversions/alternatives/* or anything else
requiring coordination among all users: the dpkg ecosystem has too many practical limitations to support non-Debian packages on anything but a
"it might work" basis (which is usually "good enough"). (This is even
true for packages within the Debian ecosystem, especially when one
considers partially implemented features like multi-arch.)
Is there any specific reason why specifically diversions are a problem
where "it might work" is not sufficient? That is, why should we divert
from the usual standard for dealing with packages outside the Debian ecosystem here?
I also caution that we've started from a very simple approach and tried fixing it up to address the problems that we recognized in analyzing it.
My impression is that we are not finished with our discovery here and
won't be for quite some time.
Well, we find limitations in dpkg that we in all other contexts usually ignore. If we used similar expectations in other cases, we would need
to very much restrict when Breaks/Conflicts/Replaces might be used at
all: it's totally unrealistic to list all (possibly local) packages
that ship conflicting files, possibly only created by maintainer
scripts. Or to explicitly list all reverse dependencies that might be
broken by a particular change. We also would not have multi-arch yet as
the dependency system doesn't support it fully (some of which is
already known, but probably discovery isn't finished yet).
(Of course in some cases explicitly listing reverse dependencies can be avoided: just always introduce something like
Provides: ${foo}-compat (= 1)
for *all* dependencies and forbid `>=` in `Depends`; this allows to
stop providing that in cases where one would have to declare explicit `Breaks` before. But only the direct provider can use this, so it's
already too limited... Alternatively forbid *all* changes that would
require this, i.e., require stable interfaces. However we do not do
this.)
But for all these issues we just say "meh, you are out of luck".
On Sun, 7 May 2023 at 06:55, Helmut Grohne <helmut@subdivi.de> wrote:
Hi Luca,
On Sat, May 06, 2023 at 09:47:15PM +0100, Luca Boccassi wrote:
Sure, there are some things that need special handling, as you have pointed out. What I meant is that I don't think we need special
handling for _all_ affected packages. AFAIK nothing is using
diversions for unit files or udev rules, for example (I mean if any package is, please point it out, because I would like a word...). I
I've posted a list in https://lists.debian.org/20230428080516.GA203171@subdivi.de and indeed, udev rules are being diverted in one case.
*fetching sledgehammer*
Is there any specific reason why specifically diversions are a problem
where "it might work" is not sufficient? That is, why should we divert
from the usual standard for dealing with packages outside the Debian ecosystem here?
Hi,
On 5/7/23 18:14, Ansgar wrote:
Is there any specific reason why specifically diversions are a problem where "it might work" is not sufficient? That is, why should we divert
from the usual standard for dealing with packages outside the Debian ecosystem here?
Locally created diversions are a supported feature, and the only way for admins to replace single files in a way that is safe for installing updates.
Even within Debian, it is not sufficient to just coordinate uploads of packages that divert each others' files, because the new diversion needs
to be in place before a newly-canonicalized package is unpacked, a
Breaks relationship does not enforce that ordering, and while a
Conflicts without a Replaces does, this adds a lot of constraints to the solver.
"Helmut" == Helmut Grohne <helmut@subdivi.de> writes:
Sure, they are supported in the sense that they can be enabled, and
then you get to keep the pieces.
But the more I think about it, the more I am convinced that the
default option working best for Debian is the one that matches the project's choice of a filesystem layout. After all, this is
configurable in the toolchain for a reason.
And the vast majority of the rest of the world has long since finished
this transition, so I struggle to think where software built with this default wouldn't work. Bullseye will be oldoldstable at that point,
and even that was default merged for new installations, and really old
ones (oldoldoldoldstable at that point? I lost count) will be long
EOL. I suppose they could still be around unmaintained, but who uses a toolchain from 8 years in the future to build software for an EOL distribution 8 years in the past? Normally it's the other way around,
as even glibc adds new symbols and is not forward compatible.
This seems somewhat convincing to me. Would you reach out to toolchain maintainers to discuss this as an early change after the release of
bookworm?
On the ELF interpreter, as long as we can reasonably ensure it works,
I do believe we should switch it, regardless of what we do with the symlinks, how we ship/add/build/package/create/manage them, as a
desired final state. Again, we should make the default in Debian work
for Debian. And given the default for Debian from Bookworm onward is
that the loader is in /usr/lib/, it seems perfectly reasonable to me
that it software built for Debian and shipped in Debian should look
there for it.
I suppose that we've been confusing the different approaches here. The question of what links base-files should contain mostly arises if you
start from the assumption that we do not modify the ELF interpreter
location. Once changing its (and /bin/sh's) location, the question of
how to install those symlinks can indeed be done in base-files.postinst
or at some other place where dpkg doesn't have to know much about it
indeed. Would you agree to examine the approach where we don't modify
the ELF interpreter location in parallel as a backup plan?
Sure, this is in the context of the ongoing discussion in the TC about revising their side of the advice.
Also, we shouldn't lose sight of the reason why this was issued in the
first place: it is designed to stop a problem from happening, and that problem can only happen when both conditions are true. I can't read
minds obviously, but I imagine that's the reason the RT advice was
worded as it was.
Hello,
On Sun 07 May 2023 at 12:03PM +01, Luca Boccassi wrote:
Sure, this is in the context of the ongoing discussion in the TC about revising their side of the advice.
I think it's highly unlikely that we revise it rather than just reissue
it, at the present time, because too many details are unsettled.
Also, we shouldn't lose sight of the reason why this was issued in the first place: it is designed to stop a problem from happening, and that problem can only happen when both conditions are true. I can't read
minds obviously, but I imagine that's the reason the RT advice was
worded as it was.
It's designed to stop as-yet-unknown problems happening, too.
Hi Luca,
On Tue, May 09, 2023 at 01:56:53AM +0100, Luca Boccassi wrote:
On Mon, 8 May 2023 at 19:06, Sean Whitton <spwhitton@spwhitton.name> wrote:
It's designed to stop as-yet-unknown problems happening, too.
Well, sure, but we've been at this for years, any such problems should really be known by now. This is with Bookworm as it stands of course,
when we go in and make more changes then we obviously have to be
careful, but that's the entire reason this thread exists and is still
going on.
This actually feels rather worrying to me. On one hand, you say that
problems should be know. On the other hand, you proposed a simple
transition with quite a number of problems that you apparently didn't
see coming. Even relatively simple mechanisms, such as just repacking
all the .debs to ship files in their canonical location and then trying
to install them, revealed a dpkg unpack error in zutils. This
combination of claiming that problems should be known while at the same
time apparently not knowing them makes me uneasy to move forward here.
So while I want to see the moratorium lifted, it all makes a lot more
sense to me given what we've seen in this thread. The worst of outcomes
I see here is the one where we cause problems that don't have a good
solution as any way forward would break someone's use case (with
someone's use case often being smooth upgrades in one way or another).
It's those where we cannot move forward nor revert.
On Mon, 8 May 2023 at 19:06, Sean Whitton <spwhitton@spwhitton.name> wrote:
It's designed to stop as-yet-unknown problems happening, too.
Well, sure, but we've been at this for years, any such problems should
really be known by now. This is with Bookworm as it stands of course,
when we go in and make more changes then we obviously have to be
careful, but that's the entire reason this thread exists and is still
going on.
On Sun 07 May 2023 at 11:14AM +02, Ansgar wrote:
Debian's dependency system requires to explicitly declare Depends/Conflicts/Replaces/Breaks, but for obvious reasons we
cannot do
that for packages outside Debian's ecosystem.
The same is true for diversions/alternatives/* or anything else
requiring coordination among all users: the dpkg ecosystem has too
many
practical limitations to support non-Debian packages on anything
but a
"it might work" basis (which is usually "good enough"). (This is
even
true for packages within the Debian ecosystem, especially when one considers partially implemented features like multi-arch.)
I don't think this is the consensus view.
Our derivatives are among our users, for example, and we care about
them being able to add packages in appropriate ways.
So why do we allow changes that require listing all reverse dependencies
in Breaks then? This is known to be wrong for all non- listed packages,
e.g., all local/vendor/derivative-specific packages.
As far as I understand, we do explicitly *not* care about our
derivatives with regard to merged-/usr as some packages in Debian
recommend users to move *away* from merged-/usr to split-/usr on
derivatives, i.e., to an unsupported fs layout.
Ansgar <ansgar@43-1.org> writes:
As far as I understand, we do explicitly *not* care about our
derivatives with regard to merged-/usr as some packages in Debian
recommend users to move *away* from merged-/usr to split-/usr on derivatives, i.e., to an unsupported fs layout.
Caring about them isn't the same thing as doing everything they want. We can both try to make things as smooth for them as possible and still make design decisions about Debian that they may disagree with or that may make some property they want to maintain difficult or impossible. It's the
sort of decision we have to make on a case-by-case basis.
On Wed, 2023-05-10 at 13:50 -0700, Russ Allbery wrote:
Caring about them isn't the same thing as doing everything they want.
We can both try to make things as smooth for them as possible and still
make design decisions about Debian that they may disagree with or that
may make some property they want to maintain difficult or impossible.
It's the sort of decision we have to make on a case-by-case basis.
Debian going out of its way to tell derivative users to switch back from merged-/usr to split-/usr is the *opposite* of trying to make things as smooth for them as possible.
Debian's dependency system requires to explicitly declare Depends/Conflicts/Replaces/Breaks, but for obvious reasons we cannot do
that for packages outside Debian's ecosystem.
The same is true for diversions/alternatives/* or anything else
requiring coordination among all users: the dpkg ecosystem has too many practical limitations to support non-Debian packages on anything but a
"it might work" basis (which is usually "good enough"). (This is even
true for packages within the Debian ecosystem, especially when one
considers partially implemented features like multi-arch.)
Ansgar <ansgar@43-1.org> writes:
Debian going out of its way to tell derivative users to switch back from merged-/usr to split-/usr is the *opposite* of trying to make things as smooth for them as possible.
Yes, I agree with that part and I think I objected to that at the time. Nonetheless, one bad decision doesn't mean that it is Debian policy that
we don't care about derivatives or their users. I think we made a mistake there which is not in alignment with our ideals or our goals. We should
try to reverse that mistake, not double down on it.
Hi,
On 26/05/2023 07:03, Ansgar wrote:
On Wed, 2023-05-10 at 14:36 -0700, Russ Allbery wrote:
Ansgar <ansgar@43-1.org> writes:
Debian going out of its way to tell derivative users to switch back from >>> merged-/usr to split-/usr is the *opposite* of trying to make things as >>> smooth for them as possible.
Yes, I agree with that part and I think I objected to that at the time.
Nonetheless, one bad decision doesn't mean that it is Debian policy that >> we don't care about derivatives or their users. I think we made a mistake >> there which is not in alignment with our ideals or our goals. We should >> try to reverse that mistake, not double down on it.
My impression is that the tech-ctte disagrees on this point and would
not want to reverse the mistake, but double down on it (in your words).
Your impression is incorrect. And assigning motivations to other parties during contentious discussions should be done with care if at all.
Consider: it is consistent to believe that it would have been better for
dpkg not to have had that warning added (quite some time ago now), but
that by now most derivatives that care will likely have patched it out
again (mitigating the harm); and if the current work on dpkg is allowed
to run its course then the warning will probably go away anyway.
So let me summarize Debian's "official" position as I understand it: we
do *NOT* care how dpkg's recommendations will break derivative installations at all; if systems become unbootable, cause data loss,
... now or in the future that is explicitly fine.
This is also unhelpful (and incorrect).
On Wed, 2023-05-10 at 14:36 -0700, Russ Allbery wrote:
Ansgar <ansgar@43-1.org> writes:
Debian going out of its way to tell derivative users to switch back from >>> merged-/usr to split-/usr is the *opposite* of trying to make things as
smooth for them as possible.
Yes, I agree with that part and I think I objected to that at the time.
Nonetheless, one bad decision doesn't mean that it is Debian policy that
we don't care about derivatives or their users. I think we made a mistake >> there which is not in alignment with our ideals or our goals. We should
try to reverse that mistake, not double down on it.
My impression is that the tech-ctte disagrees on this point and would
not want to reverse the mistake, but double down on it (in your words).
Or rather my impression is that they would like to avoid any decision
on the dpkg mess situation. (Though not making a decision when asked is
of course also an explicit decision.)
So let me summarize Debian's "official" position as I understand it: we
do *NOT* care how dpkg's recommendations will break derivative
installations at all; if systems become unbootable, cause data loss,
... now or in the future that is explicitly fine.
On Fri, 26 May 2023 at 08:39, Matthew Vernon <matthew@debian.org> wrote:
Consider: it is consistent to believe that it would have been better for
dpkg not to have had that warning added (quite some time ago now), but
that by now most derivatives that care will likely have patched it out
again (mitigating the harm); and if the current work on dpkg is allowed
to run its course then the warning will probably go away anyway.
That assumes all derivatives track unstable/testing and have taken
action, but it is possible for derivatives to track stable only, and
those would be broken.
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 546 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 58:33:57 |
| Calls: | 10,397 |
| Calls today: | 5 |
| Files: | 14,067 |
| Messages: | 6,417,461 |
| Posted today: | 1 |