[continued from previous message]

of dpkg and indeed, that kinda is the status quo where upgrades can
loose files, because dpkg has an incomplete picture of reality. The aim
of this change is to allow us to re-sync the status quo into dpkg. My
view is that dpkg's information presently is out of sync with reality
and the proposed change partially fixes that.

* This also adds undue complexity, by supporting those as admin aliases.
The admin generated redirecting symlinks are already annoying, I'd rather
not add further to that pile. I don't really want to support admins doing
this (dpkg-divert does not even support diverting a directory).

I have to agree with this one. This and the fact that this feature is
probably impossible to remove later is my main problem with the proposed change. I have been proposing it anyway, because my impression is that
it is the least bad option available. The notion of "least bad" quite
obviously is somewhat subjective and is where we need to get consensus.

[ As an aside, I think ideally eventually nothing distro provided should
be allowed to be installed within an aliased dir, and dpkg should
eventually just error out in those cases, which eventually would get
rid of the aliasing problems and any such complexity (I'm not sure how
or when that would be feasible though, but obviously in Debian at
least not until nothing ships files there). ]

It seems to me that this is something everyone agrees on. So our
disagreement resides in the way to get there rather than where to get
to.

So this still looks like a terrible interface, like it did at the time
it was discarded; founded on a hack, an interface that seems wants to
be kind of a file-type override but it cannot be, and cannot even
properly act as record tracker, etc…

I agree that in a perfect world, we would not need this. Let me circle
back to our fundamental disagreement.

My impression is that at this time basically everyone except you agrees
that we have to deal with the aliasing problems that have been rolled
out to users and will be forced in bookworm. I believe that this is the
state that we have to consider as starting point and that we cannot
magically turn this transition back to perform it in a better way. And
indeed, I believe that there would have been a better way[1] that no
longer is available to us.

On the other hand, my impression is that you continue to see the
transition as fundamentally broken and in a state that we cannot work
from. You appear to believe that if we want to do it, we must start over
in a better way. That better way must not cause aliasing problems to
dpkg.

And sure enough, DEP 17 is a result of having first created these
aliases and now trying to move the files. If we had opted for first
moving the files and then creating the aliases, much of the bad effects
would never have affected anyone and we wouldn't be discussing this
change as it would not be necessary. With DEP 17 (or any similar change
to dpkg), we will be able to actually move files to their canonical
location and thus resolve the aliasing at which point DEP 17 will become unused.

Did I accurately describe your view on this matter?

I thought it would be clear that if there is stuff that depends on
any of this kind of changes to dpkg, relying on those changes in
Debian would not be possible until after trixie+1. Of course there is
always the route to further pile up over the Jenga tower of hacks,
by for example adding huge amounts of Pre-Depends…

I agree that we probably will deal with this until at least trixie+1.
This is precisely why I would like to have a plan to finish it sooner
rather than later.

So given the above, I don't see why the apparent rush here. And as I've mentioned many times now, I'm planning to continue working on the fsys metadata stuff for 1.22.x, probably at the cost of database duplication
if necessary, if current blockers have not adapted by then. But as I've mentioned before, that might not guarantee this support is sufficient to support fixing this mess. But all other proposed changes I've seen
flying around for changes to dpkg are just conceptually wrong in one way
or another.

As I see it, the fsys metadata work may help with the aliasing problems
only if the respective symlinks are actually shipped in a data.tar of a
.deb, which is not the way we currently do things. For that reason, I
fail to see how the fsys metadata work is part of the solution for the
problems we currently experience. I'd appreciate if you could elaborate
on how you see it helping to fix the file loss on upgrade problem
affecting current installations if you have sufficient energy to do so.

Helmut

[1] What follows is to be considered a time travel fix and is purely
academic. Imagine we never had never done the usrmerge and would be
introducing it again in a usrmerge package that works quite
differently. Rather than create those aliases upfront, it would
declare a trigger interest in affected locations (i.e. both /bin and
/usr/bin and so on). It would practically trigger on every package
operation. Whenever triggered, it would scan the non-/usr location
for files. If any are found, it would populate both the /usr
location and the non-/usr location with symlinks to the actual
files. However when the non-/usr location becomes empty, it would
remove the hierarchy and place the symlink directly. In that
approach, we'd end up at the same layout that we currently have, but
use symlink farms until we reach the point that all files have been
moved. There are a number of aspects where this approach is
problematic as well, but I think it is no longer useful to discuss
it as switching to this approach is not an available option anymore.

--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)

[continued from previous message]

I'd also be interested on how you plan to move important files in essential packages. This is an aspect raised by Simon Richter and where
I do not see an obvious answer yet.

Do you have a pointer? Not sure I follow what "important" files means
here, doesn't ring a bell.

In <669234b3-555b-4e2a-ccc7-dd5510b6e9c1@debian.org>, Simon Richter
said:

Dpkg already has defined behaviour for directory vs symlink: the directory wins. In principle a future version of dpkg could change that, but /lib/ld-linux.so.2 is just too special, we'd never want to have a package that
actually moves it.

This and /bin/sh is the kind of files I'd consider important. And then
upon thinking further it became more and more difficult for me to make
sense of the objection. On a merged system, we can just move that file
to its canonical location without having any trouble even with an
unmodified dpkg. So from my pov, the question about important files can
be disregarded. I hope Simon Richter agrees.

Let us circle back to your "broken" approach. It sure is simple (just
move all the files and be done) and if we could just skip over the
upgrade issues and have all the files moved without having to modify
dpkg, that would actually be a better result than DEP 17. Just how do we avoid the issue of file loss arising from the aliasing in your scenario?

There kinda is an obvious solution here. We just need to tell dpkg that
it needs to remove the package containing the file that is being moved
before unpacking the replacing package. This semantic actually exists
and we call it "Conflicts". So instead of using Breaks+Replaces, the
solution is to use Conflicts! Problem solved, right?

Yeah, I think this solves a number of cases, but there are two areas
where it does not:
* We generally prefer Breaks over Conflicts for a reason. It gives the
dependency resolver more freedom and in taking this freedom away, it
may fail to find solutions to complex upgrades. (Which is why Breaks
got introduced in the first place.)
* We cannot use Conflicts inside the transitively essential set.

So if we move all those files, in 90% (number made up) of the cases it
will go well (since we don't move between packages) and in 90% (number
made up) of the remaining 10%, we can use Conflicts, but what do we do
about that remaining 1%?

If we look deeper into the dpkg toolbox, we pause at diversions. What if
the new package were to add a (--no-rename) diversion for files that are
at risk of being accidentally deleted in newpkg.preinst and then remove
that diversion in newpkg.postinst? Any such diversion will cause package removal of the oldpkg to skip removal of the diverted file (and instead deleted the non-existent path that we diverted to). Thus we retain the
files we were interested in.

This way, we can just move the files as you suggested.
* For 90% of the packages, this will just work.
* For 9% of the packages, we'll need to turn Breaks+Replaces into
Conflicts.
* And for 1% of the packages, we'll need to add complex diversions to
maintainer scripts.

And while this sounds super ugly in the 1% of cases, it's a complexity
that maybe isn't necessary at all and we can remove it after trixie
unlike the complexity being added in DEP 17.

In order to better understand the mechanics at hand (and due to Simon Richter's call for test cases), I sat down and wrote some. So in this
mail you find 4 files attached:
* runtest.sh is a wrapper script to run each case inside a fresh
chroot created by mmdebstrap (as you don't want to mess your system).
* case1.sh demonstrates the file loss problem with Breaks+Replaces and
thus fails.
* case2.sh demonstrates how Conflicts fix the problem.
* case3.sh demonstrates how diversions fix the problem.

In sincerely hope that this fixed-up plan doesn't have any serious
issues. If you find any please tell.

And before closing this mail, I would like to express my gratitude and
thanks to Emilio Pozuelo Monfort for enduring me in numerous discussions
on this matter and providing so many of the key insights captured in
this mail.

Brilliant! Would never have thought of using divert like that.

So, what work would need to happen to make this reality? Do we need tooling/scripts/build changes to support the divert scheme, or is it
"simply" a matter of documenting and testing?

Kind regards,
Luca Boccassi

--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)

[continued from previous message]

Yeah, I think this solves a number of cases, but there are two areas
where it does not:
* We generally prefer Breaks over Conflicts for a reason. It gives the
dependency resolver more freedom and in taking this freedom away, it
may fail to find solutions to complex upgrades. (Which is why Breaks
got introduced in the first place.)
* We cannot use Conflicts inside the transitively essential set.

So if we move all those files, in 90% (number made up) of the cases it
will go well (since we don't move between packages) and in 90% (number
made up) of the remaining 10%, we can use Conflicts, but what do we do
about that remaining 1%?

If we look deeper into the dpkg toolbox, we pause at diversions. What if
the new package were to add a (--no-rename) diversion for files that are
at risk of being accidentally deleted in newpkg.preinst and then remove
that diversion in newpkg.postinst? Any such diversion will cause package removal of the oldpkg to skip removal of the diverted file (and instead
deleted the non-existent path that we diverted to). Thus we retain the
files we were interested in.

This way, we can just move the files as you suggested.
* For 90% of the packages, this will just work.
* For 9% of the packages, we'll need to turn Breaks+Replaces into
Conflicts.
* And for 1% of the packages, we'll need to add complex diversions to
maintainer scripts.

And while this sounds super ugly in the 1% of cases, it's a complexity
that maybe isn't necessary at all and we can remove it after trixie
unlike the complexity being added in DEP 17.

In order to better understand the mechanics at hand (and due to Simon
Richter's call for test cases), I sat down and wrote some. So in this
mail you find 4 files attached:
* runtest.sh is a wrapper script to run each case inside a fresh
chroot created by mmdebstrap (as you don't want to mess your system).
* case1.sh demonstrates the file loss problem with Breaks+Replaces and
thus fails.
* case2.sh demonstrates how Conflicts fix the problem.
* case3.sh demonstrates how diversions fix the problem.

In sincerely hope that this fixed-up plan doesn't have any serious
issues. If you find any please tell.

And before closing this mail, I would like to express my gratitude and
thanks to Emilio Pozuelo Monfort for enduring me in numerous discussions
on this matter and providing so many of the key insights captured in
this mail.

Helmut

#!/bin/shset -euxTESTCASE="$1"MIRROR="${2:-http://deb.debian.org/debian}"mmdebstrap --hook-dir=/usr/share/mmdebstrap/hooks/merged-usr unstable /dev/null "$MIRROR" --variant=apt --include=dpkg-dev --customize-hook="upload '$TESTCASE' /testcase" --
customize-hook='chroot "$1" sh /testcase'
#!/bin/shset -euxmkdir -p pkga/DEBIAN pkga/libcat >pkga/DEBIAN/control <<EOFPackage: pkgaArchitecture: allVersion: 0Maintainer: none@noneDescription: pkgaEOFtouch pkga/lib/canarydpkg-deb -b pkgamkdir -p pkgb/DEBIAN pkgb/usr/libcat >pkgb/
DEBIAN/control <<EOFPackage: pkgbArchitecture: allVersion: 0Breaks: pkgaReplaces: pkgaMaintainer: none@noneDescription: pkgbEOFtouch pkgb/usr/lib/canarydpkg-deb -b pkgbdpkg -i pkga.debstat /usr/lib/canarydpkg --auto-deconfigure --unpack
pkgb.debstat /usr/lib/canary || :dpkg -r pkgastat /usr/lib/canary || :dpkg --configure -astat /usr/lib/canary
#!/bin/shset -euxmkdir -p pkga/DEBIAN pkga/libcat >pkga/DEBIAN/control <<EOFPackage: pkgaArchitecture: allVersion: 0Maintainer: none@noneDescription: pkgaEOFtouch pkga/lib/canarydpkg-deb -b pkgamkdir -p pkgb/DEBIAN pkgb/usr/libcat >pkgb/
DEBIAN/control <<EOFPackage: pkgbArchitecture: allVersion: 0Conflicts: pkgaMaintainer: none@noneDescription: pkgbEOFtouch pkgb/usr/lib/canarydpkg-deb -b pkgbdpkg -i pkga.debstat /usr/lib/canaryapt-get -y install ./pkgb.debstat /usr/lib/
canary
#!/bin/shset -euxmkdir -p pkga/DEBIAN pkga/libcat >pkga/DEBIAN/control <<EOFPackage: pkgaArchitecture: allVersion: 0Maintainer: none@noneDescription: pkgaEOFtouch pkga/lib/canarydpkg-deb -b pkgamkdir -p pkgb/DEBIAN pkgb/usr/libcat >pkgb/
DEBIAN/control <<EOFPackage: pkgbArchitecture: allVersion: 0Maintainer: none@noneDescription: pkgbEOFcat >pkgb/DEBIAN/preinst <<EOF#!/bin/shdpkg-divert --add --no-rename --divert /lib/canary.usrmerged /lib/canaryEOFcat >pkgb/DEBIAN/postinst <<
EOF#!/bin/shdpkg-divert --remove --no-rename /lib/canaryEOFchmod +x pkgb/DEBIAN/preinst pkgb/DEBIAN/postinsttouch pkgb/usr/lib/canarydpkg-deb -b pkgbdpkg -i pkga.debstat /usr/lib/canarydpkg --auto-deconfigure --unpack pkgb.debstat /usr/lib/
canarydpkg -r pkgastat /usr/lib/canarydpkg --configure -astat /usr/lib/canary

--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)

[continued from previous message]

move all the files and be done) and if we could just skip over the
upgrade issues and have all the files moved without having to modify
dpkg, that would actually be a better result than DEP 17. Just how do we avoid the issue of file loss arising from the aliasing in your scenario?

There kinda is an obvious solution here. We just need to tell dpkg that
it needs to remove the package containing the file that is being moved
before unpacking the replacing package. This semantic actually exists
and we call it "Conflicts". So instead of using Breaks+Replaces, the
solution is to use Conflicts! Problem solved, right?

Yeah, I think this solves a number of cases, but there are two areas
where it does not:
* We generally prefer Breaks over Conflicts for a reason. It gives the
dependency resolver more freedom and in taking this freedom away, it
may fail to find solutions to complex upgrades. (Which is why Breaks
got introduced in the first place.)
* We cannot use Conflicts inside the transitively essential set.

So if we move all those files, in 90% (number made up) of the cases it
will go well (since we don't move between packages) and in 90% (number
made up) of the remaining 10%, we can use Conflicts, but what do we do
about that remaining 1%?

If we look deeper into the dpkg toolbox, we pause at diversions. What if
the new package were to add a (--no-rename) diversion for files that are
at risk of being accidentally deleted in newpkg.preinst and then remove
that diversion in newpkg.postinst? Any such diversion will cause package removal of the oldpkg to skip removal of the diverted file (and instead deleted the non-existent path that we diverted to). Thus we retain the
files we were interested in.

This way, we can just move the files as you suggested.
* For 90% of the packages, this will just work.
* For 9% of the packages, we'll need to turn Breaks+Replaces into
Conflicts.
* And for 1% of the packages, we'll need to add complex diversions to
maintainer scripts.

Do we have a tool to classify the packages ?

Bastien

--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)

[continued from previous message]

* mailagent: comment about diversions
* oping: checks for an existing diversion
* psgml: comment about diversions
* ucf: comment about diversions
* wireshark-common: checks for an existing diversion

So yeah, with the exception of dash, this looks fairly good. Let me also
dive into dash. Unlike the majority of diverters, it diverts in postinst
rather than preinst to allow controlling /bin/sh via debconf. A similar technique is in effect by gpr. In any case, this is special, because
dash diverts its own files, so when moving dash's file, its diversions
can be migrated at the same time. It merely means, that we cannot have debhelper just move files (as that would horribly break dash) and
instead have to move files on a package-by-package way. We could also
opt for removing dash's diversion in the default case and there even is
a patch for doing so (#989632) since almost two years. Too bad we didn't
apply it. In any case, as long as the file moving is not forced via
debhelper, dash should be harmless.

With this number, another option is on the table. Rather than divert dpkg-divert, we could just fix these 8 packages to duplicate their
diversions for /usr and then when moving the underlying files add
versioned Conflicts to the old version of diverters (none of which are essential). So this is an order of 15 uploads (8 diverters, 6 diverted packages, dash).

Luca Boccassi kindly pointed me at config-package-dev though. This is a
tool for generating local packages and it also employs dpkg-divert.
There is a significant risk of breaking this use case. If we were to
divert dpkg-divert and automatically duplicate diversions, this use case
were automatically covered.

I am unsure how to proceed here and request assistance from the
debathena project to evaluate the situation. If possible, I'd like to
avoid the complexity of wrapping dpkg-divert.

Helmut

--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)

[continued from previous message]

diversion of /bin/zfgrep to /bin/zfgrep.gzip by zutils in stable, testing, unstable
diversion of /bin/zgrep to /bin/zgrep.gzip by zutils in stable, testing, unstable

All other diversion affect /etc or /usr and I think we're not going to
move any files from /usr to /. So this is a complete list as of today
and I have to say, I expected it to be longer. In effect, we're talking
about merely 8 packages.

For completeness sake, I also looked at the other packages mentioning dpkg-divert in their preinst to catch false negatives. I'll skip
diversions inside /usr as well as removals of diversions here:
* amazon-ec2-net-utils: diversion inside /etc
* angband: comment about diversions
* arpwatch: comment about diversions
* dash: complex use of conditional diversions via postinst
* dist: comment about diversions
* gpr: conditional diversion (inside /usr)
* iputils-arping: check for an existing diversion
* iputils-clockdiff: check for an existing diversion
* iputils-ping: check for an existing diversion
* ld10k1: comment about diversions
* mailagent: comment about diversions
* oping: checks for an existing diversion
* psgml: comment about diversions
* ucf: comment about diversions
* wireshark-common: checks for an existing diversion

This is great data - indeed, 8 packages seems eminently fixable.

So yeah, with the exception of dash, this looks fairly good. Let me also
dive into dash. Unlike the majority of diverters, it diverts in postinst rather than preinst to allow controlling /bin/sh via debconf. A similar technique is in effect by gpr. In any case, this is special, because
dash diverts its own files, so when moving dash's file, its diversions
can be migrated at the same time. It merely means, that we cannot have debhelper just move files (as that would horribly break dash) and
instead have to move files on a package-by-package way. We could also
opt for removing dash's diversion in the default case and there even is
a patch for doing so (#989632) since almost two years. Too bad we didn't apply it. In any case, as long as the file moving is not forced via debhelper, dash should be harmless.

If I understand correctly, by "forced via debhelper" you mean the
proposal of fixing the paths at build time, right? But if not via
that, it means having to fix all of them by hand, which is a lot - is
it possible to fix dash instead? Or else, we could add an opt-out via
one of the usual dh mechanisms, and use it only in dash perhaps?

With this number, another option is on the table. Rather than divert dpkg-divert, we could just fix these 8 packages to duplicate their
diversions for /usr and then when moving the underlying files add
versioned Conflicts to the old version of diverters (none of which are essential). So this is an order of 15 uploads (8 diverters, 6 diverted packages, dash).

Luca Boccassi kindly pointed me at config-package-dev though. This is a
tool for generating local packages and it also employs dpkg-divert.
There is a significant risk of breaking this use case. If we were to
divert dpkg-divert and automatically duplicate diversions, this use case
were automatically covered.

I am unsure how to proceed here and request assistance from the
debathena project to evaluate the situation. If possible, I'd like to
avoid the complexity of wrapping dpkg-divert.

Which part of config-package-dev causes a conflict here? Is it
something that can be fixed? Given it's declarative, an upload + rdeps
rebuild should be all that's needed, assuming we know what the issue
is and how to fix it. As far as I can remember, it's a build-time
utility and everything it does is embedded in the target package's
maintainer scripts. But it's been a few years since I last used it, so
I might remember wrongly.

Kind regards,
Luca Boccassi

--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)

[continued from previous message]

target would have to be removed. When doing that, user selection is
probably lost and in a similar way, we break users who do this
programmatically (e.g. ansible/puppet). So best leave this as is
eternally.

3. Link targets for slave links. As far as I can see, we can just
change these and on the next package update running
update-alternatives --install will work despite printing

update-alternatives: warning: forcing reinstallation of alternative ... because link group pager is broken

None of these seems convertible at ease and the third category has
basically no items, so I think we should probably just do nothing about alternatives and let them continue to use non-canonical paths. As far as
I can see, that should just work. Does anyone disagree with that
judgement?

Helmut

--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)

[continued from previous message]

iQJNBAEBCgA3FiEEm5FwB64DDjbk/CSLaVt65L8GYkAFAmRW5m4ZHHNwd2hpdHRv bkBzcHdoaXR0b24ubmFtZQAKCRBpW3rkvwZiQPXpD/0TD9vGmxYKzttI5RsoYvXL lbho6P5uIT9xRVUtflk3LuGupbhSIoEeQYeYA2oASd2M4TrG6f4oIJgeBomPh5Zt vhbB4g0XmJ4n8lytZUlBd+kFT+gT/bnmaUQoqRbXnfgekA9Egn4saarnKfQ261Gy mCxy7a6EX6mFckZM3Qsx4ZaWmA8ardWX24q4+F8OOvGRwQmrzk4s6RRZOy1fvphG ca5woq9+2aQS/xXJ20QNBgHQa3oJrPq6xnWjq1aN+Q/orsS1Qohj+rR4O8nXUtIG vnE+YAwhZntjAd8S9EYXJhrHibas7Gax3wn1IijUvD9gG9KNW7CMs5+sB4UiFyg3 uJZh1B2/GWDro6QSco167ezB5I7HkoEJyl0A5xYX4siyY/ql54rA5Zmp/bvhTuJB KuDCnx/er3VRl12E8UmfvYXB39tDx+sPEaVt1Ipr0/YsxTi3diT9b+BauGrP+/v0 tl2beGVXtJxpMHB3afV/9oyRUqFohqGTZWc6CrKmOwL63HyyUx9sn/VBBEr9qL4p X9SYRikNqwKg7Db+Ps//K2nyA6uU58/YTCJMLKDSX/FbDfZLvzbEpMKe0APZEiVw dvK88jRHZczQa/A6G7VRoYYxd6UrTBkfd26fsgQof4z1D/zRg0LSqc42aoyX8kEM E9liMDEnlumOCzffmVEjIw==2Ty3
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)

[continued from previous message]

but I don't think we should impose this workflow with pre-depends and diversions for all affected packages. I think it should be mandatory
for problematic packages such as those you already pointed out, _and_
for cases where the maintainer wants to move files also between binary packages.

5. Write a policy on how to perform changes and how to move files. Simon
Richter suggested a policy for dpkg-divert already. This needs to be
extended to cover other matters and refined.

6. Convert packages one by one by enabling the addon or bumping the
compat level. Such a conversion may require:
* Adding Pre-Depends: ${misc:Pre-Depends}
* Adding Conflicts
* Adding invocations of the usr-merge-helper
* Modifying maintainer scripts to canonicalize diversions
* ...

As above.

7. Continuously monitor the reports from the service and turn that
feedback into patches.

This plan definitely is incomplete and misses critical steps. I suspect
that Simon Richter could immediately tell at least two flaws if not
more.

A rather obvious flaw (that Simon Richter already mentioned) is how this breaks filesystem bootstrap. The policy requirement is that all
essential packages must work in unpacked state provided that they have
been configured at least once. (Please don't ask who added this
requirement to policy. I might be embarrassed.) In reality bootstrap implementations expect more. In order to perform that initial
configuration, we expect that they work sufficiently well to be able to
run maintainer scripts even in unpacked state prior to initial
configuration. This currently works due to the dynamic linker being
shipped in the non-canonical path it is referenced as. Moving it will
break bootstrap tooling unless we install /lib in some binary package
(as Simon Richter pointed out). However, due to how dpkg handles
directory vs symlink conflicts, it is not clear whether we can
reasonably add it as a symlink to any data.tar without causing havoc.
This definitely needs more research and is one of the areas where
teaching dpkg about aliasing brings significant benefits.

Curve ball question: is there anything blocking us from canonicalizing PT_INTERP at the source in gcc so that the essential set (and
everything else, really) can work without the lib -> usr/lib symlink?
Would anything obvious break?
This would be in trixie, so even when unpacked on bookworm for the
upgrade case, the loader is guaranteed to be accessible from the
canonical path. Heck, we could even consider canonicalizing shebangs
in scripts shipped in essential packages? I'd imagine /bin/sh would
have the same issue as the loader?

Let me restate that I am still not convinced that we can pull this stunt without painting us in a very dark corner. Too many moving parts of this approach do not have well-understood solutions. And even then, my
analysis has mainly focused on packages shipped by Debian. It neglected
other relevant aspects such as:
* custom Debian packages (e.g. equivs, config-package-dev, private
packages)
* vendor packages
* local diversions
* local statoverrides
* probably more

I don't think we should lose sleep over third party/proprietary
packages. Given most of those are autogenerated from dubious scripts
(I have seen things...) I doubt they are using diverts and such, most
packages I see from companies are built using cmake or java built-in
scripts that shove stuff in /opt and manually generate the .ar and
pack them, so...
I mean, we were not concerned at all when Canonical switched to zstd
for deb compression, creating a massive ticking time bomb given the
vast majority of third party .debs are built on some flavour of Ubuntu
and ship a single .deb for all consumers, and thus would stop being
installable on Debian (because for years the dpkg patch was rejected, thankfully now it's all sorted and we are good for bookworm), I don't
think we should spend time trying to pre-empt fixing those. Noting in
the release notes that in case they do X and Y and Z they will need
adjustments and pointing to appropriate documentation seems more than
enough to me, no?
The most popular ones (by virtue of the very fair and impartial
property of being currently installed on my laptop) like chrome,
vscode, edge, steam and signal do not ship anything in the legacy
directories anyway.

Same for local modifications, documenting in the release notes what
corner cases people need to be aware of and how to deal with them
seems standard practice to me when doing new releases. For example
there's a bunch of things you were supposed to do on your machine when upgrading to Bullseye as documented at: https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#upgrade-specific-issues
and without checking, I'm pretty sure this is normal for all releases.

Kind regards,
Luca Boccassi

--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)

[continued from previous message]

This would be in trixie, so even when unpacked on bookworm for the
upgrade case, the loader is guaranteed to be accessible from the
canonical path. Heck, we could even consider canonicalizing shebangs
in scripts shipped in essential packages? I'd imagine /bin/sh would
have the same issue as the loader?

Heh! As much as this initially sounds insanely crazy, I was unable to
spot a reason for why this cannot work at all. It definitely sounds like
fixing the bootstrap protocol to me. On the flip side, I'd assume the
loader location (for the purpose of examining ELF objects) to be hard
coded in quite some places. Changing gcc seems out of question, because
we also expect binaries built on Debian to run on non-Debian systems.
Rather than change gcc, I think the interpreter could be changed
after-the-fact using patchelf. We do have prior art for this since GUIX
uses this technique to redirect the loader below /gnu/store to implement
its versioning scheme and allow concurrent installations of glibc. The existence of GUIX should also limit the possible breakage. By not doing
it in gcc, we likely save apparmor's and heaptrack's test suites from
breaking. Still radare2 an systemtap seem fragile in this regard. (You
can use codesearch.d.n with lib64/ld-linux-x86 and one of those packages
to discover the code I see breaking. My search was non-exhaustive.)

How about the long-term vision of this? Elsewhere you indicated that
you'd like the aliasing symlinks to not be shipped by any data.tar. Does
that imply that we'd keep patching the interpreter and using /usr/bin/sh forever in the essential set? If adding the links to base-files, it
would be of temporary nature only.

If adding the symlinks to base-files, how about /lib64? Would we ship it
for all architectures or just for those that need it (e.g. amd64,
loong64, mips64el, ppc64, ppc64el)? https://wiki.debian.org/ArchitectureSpecificsMemo has a list of dynamic
loaders we also need /libx32 for x32 at least. If making this architecture-dependent, would base-files become Multi-Arch: same?

I don't think we should lose sleep over third party/proprietary
packages. Given most of those are autogenerated from dubious scripts
(I have seen things...) I doubt they are using diverts and such, most packages I see from companies are built using cmake or java built-in
scripts that shove stuff in /opt and manually generate the .ar and
pack them, so...
I mean, we were not concerned at all when Canonical switched to zstd
for deb compression, creating a massive ticking time bomb given the
vast majority of third party .debs are built on some flavour of Ubuntu
and ship a single .deb for all consumers, and thus would stop being installable on Debian (because for years the dpkg patch was rejected, thankfully now it's all sorted and we are good for bookworm), I don't
think we should spend time trying to pre-empt fixing those. Noting in
the release notes that in case they do X and Y and Z they will need adjustments and pointing to appropriate documentation seems more than
enough to me, no?

What you write here makes it obvious that we do not have consensus on
this aspect. I caution that those not interested in /usr-merge would
rather have things continue to just work even if dubious, so it seems to
me that worrying a bit more than we would ourselves would increase the
chances of our transition to be socially acceptable.

The most popular ones (by virtue of the very fair and impartial
property of being currently installed on my laptop) like chrome,
vscode, edge, steam and signal do not ship anything in the legacy
directories anyway.

I vaguely agree that vendor packages are the category with most unlikely breakage here.

Same for local modifications, documenting in the release notes what
corner cases people need to be aware of and how to deal with them
seems standard practice to me when doing new releases. For example
there's a bunch of things you were supposed to do on your machine when upgrading to Bullseye as documented at: https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#upgrade-specific-issues
and without checking, I'm pretty sure this is normal for all releases.

It is pretty normal that people don't read release-notes even though
they should. The approach of "let's put it in release notes and be done"
also fails on another level. For instance, both live-build and open-infrastructure-system-tools employ a local diversion of
/bin/hostname as part of their image building implementation.

At this time, I am strongly convinced that this forced file move
strategy to resolving the /usr-merge very much is a game of
whack-a-mole, which is not meant as ruling out the option, because all
known strategies have significant downsides.

Helmut

--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)

[continued from previous message]

'worry' needs to be realistic (eg: a conflict due to /bin/foo and
/usr/bin/foo both existing is not something that needs to cause any
worrying, as it is already disallowed by policy and unsupported), and
congruent with usual practices that are applied for all other changes
and processes. IE, the appropriate weighting should be given to
potential problems spotted.

Kind regards,
Luca Boccassi

--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)

[continued from previous message]

I don't think this is true? At least not in the broader sense: if you
compile something on Debian, it will obviously get linked against
libraries and dependencies as they are in Debian.
Perhaps what you mean is that, given an entire separate sysroot-like
tree, passing the appropriate compiler and linker flags and
environment variables, you can use the local compiler we ship to build 'foreign' programs. That is true, but again it requires to set up the environment appropriately, including linker flags. And the caller
needs to ensure the environment, including linker flags, is
appropriate for the target environment (I guess 'host' environment, in
GNU parlance). Therefore, I don't think it would be unreasonable to
require that if the target environment is split-usr, then the caller
also needs to specify an appropriate
'-Wl,--dynamic-linker=/lib/ld-whatever' option.

I do not want to break use cases, but at the same time I don't think
it's fair to impose that any and all possible use cases have to
forever work without any changes - that's not how we approach building
new major releases. We very much allow ABI breaking changes, to pick
the most obvious example, and we rebuild our stuff but leave it up to
the users to rebuild their own local programs to deal with it.

Or to put it in another way: I think our defaults should prioritize
the Debian native use case. Given we ship our loader in /usr/lib/ld*
now, it makes sense to me that the default in GCC is to point to
/usr/lib/ld*. Callers can override that as needed for third-party/external/foreign use cases.

It is important to note, just to be clear, that we will most likely
forever have the lib -> usr/lib symlinks in the final image when it is booted/instantiated/etc, so existing software that uses the
non-canonicalized PT_INTERP will continue to work normally.

Rather than change gcc, I think the interpreter could be changed after-the-fact using patchelf. We do have prior art for this since GUIX
uses this technique to redirect the loader below /gnu/store to implement
its versioning scheme and allow concurrent installations of glibc. The existence of GUIX should also limit the possible breakage. By not doing
it in gcc, we likely save apparmor's and heaptrack's test suites from breaking. Still radare2 an systemtap seem fragile in this regard. (You
can use codesearch.d.n with lib64/ld-linux-x86 and one of those packages
to discover the code I see breaking. My search was non-exhaustive.)

How about the long-term vision of this? Elsewhere you indicated that
you'd like the aliasing symlinks to not be shipped by any data.tar. Does
that imply that we'd keep patching the interpreter and using /usr/bin/sh forever in the essential set? If adding the links to base-files, it
would be of temporary nature only.

If adding the symlinks to base-files, how about /lib64? Would we ship it
for all architectures or just for those that need it (e.g. amd64,
loong64, mips64el, ppc64, ppc64el)? https://wiki.debian.org/ArchitectureSpecificsMemo has a list of dynamic loaders we also need /libx32 for x32 at least. If making this architecture-dependent, would base-files become Multi-Arch: same?

As above, I am pretty sure the running system will forever have the
bin -> usr/bin, lib -> usr/lib etc. symlinks, so the vast majority of
external programs should be unaffected? IE if you try to search
manually for ld, you'll still find it at the old path. And if there is hard-coded parsing of ELF sections anywhere then yeah it will need to
be rebuilt, but to me it seems like any other ABI changes.

I think we should leave the long term vision for another day, and
focus on your requirements for the essential set unpacking right now.

Kind regards,
Luca Boccassi

--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)

[continued from previous message]

Knowing the target state of a transition seems fairly fundamental to implementing it and base-files is part of the essential set. To me, it
is a significant difference whether we temporarily or permanently modify
the ELF interpreter in the essential set. For these reasons, I do think
the answers to these questions do matter at this time. As long as we do
not have answers here, we must not move ld.so nor /bin/sh regardless of
whether we patch dpkg or not.

Helmut

--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)

[continued from previous message]

handling and no diversion, then it should be simple to handle: the debhelper in stable will not perform the conversion by definition as
the logic won't be present, and any dh upload to backports will have
such logic disabled, so that other packages that get uploaded to
backports and built with either the stable or the backports debhelper
won't have any change performed on them.

As much a I'd like to trust you on things actually being simple, we've
seen over and over again that the simple approaches have non-trivial
flaws. If you were to highlight resulting problems (and propose
solutions), that would be more convincing to me than continuously
labeling it simple.

Or to put it in another way: I think our defaults should prioritize
the Debian native use case. Given we ship our loader in /usr/lib/ld*
now, it makes sense to me that the default in GCC is to point to /usr/lib/ld*. Callers can override that as needed for third-party/external/foreign use cases.

I guess you'll be having a hard time convincing the toolchain
maintainers of this change, but my other point was that this is
unnecessary when we can use patchelf after the fact.

Sure that is possible, you can manually patch the essential set by
hand, and that would solve your requirement. And that's certainly a
viable fallback avenue to keep around.

But the more I think about it, the more I am convinced that the
default option working best for Debian is the one that matches the
project's choice of a filesystem layout. After all, this is
configurable in the toolchain for a reason.
And the vast majority of the rest of the world has long since finished
this transition, so I struggle to think where software built with this
default wouldn't work. Bullseye will be oldoldstable at that point,
and even that was default merged for new installations, and really old
ones (oldoldoldoldstable at that point? I lost count) will be long
EOL. I suppose they could still be around unmaintained, but who uses a toolchain from 8 years in the future to build software for an EOL
distribution 8 years in the past? Normally it's the other way around,
as even glibc adds new symbols and is not forward compatible.

How about the long-term vision of this? Elsewhere you indicated that you'd like the aliasing symlinks to not be shipped by any data.tar. Does that imply that we'd keep patching the interpreter and using /usr/bin/sh forever in the essential set? If adding the links to base-files, it
would be of temporary nature only.

If adding the symlinks to base-files, how about /lib64? Would we ship it for all architectures or just for those that need it (e.g. amd64, loong64, mips64el, ppc64, ppc64el)? https://wiki.debian.org/ArchitectureSpecificsMemo has a list of dynamic loaders we also need /libx32 for x32 at least. If making this architecture-dependent, would base-files become Multi-Arch: same?

...

I think we should leave the long term vision for another day, and
focus on your requirements for the essential set unpacking right now.

Knowing the target state of a transition seems fairly fundamental to implementing it and base-files is part of the essential set. To me, it
is a significant difference whether we temporarily or permanently modify
the ELF interpreter in the essential set. For these reasons, I do think
the answers to these questions do matter at this time. As long as we do
not have answers here, we must not move ld.so nor /bin/sh regardless of whether we patch dpkg or not.

On the ELF interpreter, as long as we can reasonably ensure it works,
I do believe we should switch it, regardless of what we do with the
symlinks, how we ship/add/build/package/create/manage them, as a
desired final state. Again, we should make the default in Debian work
for Debian. And given the default for Debian from Bookworm onward is
that the loader is in /usr/lib/, it seems perfectly reasonable to me
that it software built for Debian and shipped in Debian should look
there for it.

Kind regards,
Luca Boccassi

--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)

[continued from previous message]

appropriate version of dpkg, and/or its use disallowed until trixie+1
for the convenience of backporters.

Simon

--- SoupGate-Win32 v1.05
* Origin: you cannot sedate... all the things you hate (1:229/2)