Hello fellow developers,

I have bad news about /usr-move. The primary mitigation (M18) for
aliased diversions (P3) as implemented by me in quite a few packages is
broken. This was brought to my attention by independent reports of Emil Södergren and Colin Watson on live-build and debian-installer-utils respectively. I now have a better understanding of dpkg-divert (when it
is faced with aliasing) and need to redo most patches (probably about
20). Affected packages shall receive updated patches during January. I'm
sorry for the inconvenience caused.

If your have a lot of time and are interested in the underlying
technical details, I invite you to continue reading. Otherwise, consider
doing something more pleasant.

Fundamentally, I assumed that dpkg-divert --rename would only move a
file if it was considered installed according to the dpkg database. What
really happens is that --rename moves a file as long as it exists. In
many cases that I tested, this would happen to work by chance, but
important corner cases would be wrong. We start with an aliased file
being diverted. M18 says that the diversion should be duplicated to the canonical location. Unfortunately, we cannot translate the diversion
target in the same way. Say we move /bin/foo to /usr/bin/foo in a
package upgrade. dpkg will first put all the new files into temporary locations, then move them all into place (including /usr/bin/foo) and
then unlink any removed files (including /bin/foo). Due to a sanity
check, it notices that /bin/foo was just unpacked and skips its
deletion. This sanity check is not in effect when a file is diverted. If
we were to alias the diversion targets, the diverted file would be
reliably lost. This is why all of the duplicated diversions use a .usr-is-merged suffix. This brings another consequence. When diverting,
we must rename the file depending on whether it is aliased or canonical,
but that's not what --rename does. No matter how we order the
dpkg-divert calls it will wrongly rename either the aliased or the
canonical case. And that's what we've seen in debian-installer-utils and live-build.

As a result, I now propose using dpkg-divert with --no-rename for
duplicated diversions and doing the move manually consulting dpkg -S to
figure out where to move to. Beware that dpkg-divert is normally
idempotent and skips a rename when a file already is diverted, so as we implement the rename externally, it must properly be conditional to the diversion being newly added to retain idempotency.

Since I was failing to get this right earlier, I figured I could as well
write some tests. As a result, I'm attaching a shell script. It
constructs a few demo packages and attempts a hopefully exhaustive
combination of using them and dpkg-divert by writing its experiments to
the working directory. Running it should be fairly boring. If you happen
to review it, I appreciate a response.

The newly proposed logic becomes fairly elaborate. As there is no place
that could share it and we want to get rid of it fairly soon, I intend
to copy the code into the relevant users.

A slight simplification can be achieved if the diverting package depends
on a version of the diverted package that has canonicalized the
diverted file. Then we may use --rename on the canonical path and
--no-rename on the aliased path. As a result, adding the diversion in
preinst may wrongly rename the file to the canonical diversion target,
but the dependency ensures that it'll be clobbered with an updated
version. Once the dependency is unpacked, this error is papered over.

codesearching DEP17.*M18 (and a few other patterns) suggest the
following packages needing an update:
* bfh-metapackages
* clonezilla
* cryptsetup-nuke-password (and cryptsetup)
* debian-installer-utils
* isc-dhcp
* molly-guard
* live-build
* open-infrastructure-system-tools
* opensysusers
* piuparts
* progress-linux-metapackages
* systemd
* zutils (and gzip)

I've not reviewed them yet. debian-installer-utils has a fixed MR and live-build definitely is broken. Please expect me to follow up on each
package within January.

Helmut

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Jan 08, 2025 at 03:21:43PM +0100, Helmut Grohne wrote:

write some tests. As a result, I'm attaching a shell script. It

A kind soul reminded me of actually attaching something.

Helmut

#!/bin/sh# shellcheck disable=SC2317 # Our way of running test cases confuses shellcheck.set -e# Create three Debian packages:# * m18base.deb emulates base-files and installs /lib and /usr/lib.# * m18dummy_1.deb installs /lib/m18dummy.# *
m18dummy_2.deb installs /usr/lib/m18dummy.rm -Rf root m18base m18dummyrm -f m18base.deb m18dummy_1.deb m18dummy_2.debmkdir -p m18base/DEBIAN m18base/usr/libln -s usr/lib m18base/libcat >m18base/DEBIAN/control <<EOFPackage: m18baseArchitecture:
allEssential: yesVersion: 1Maintainer: Dummy <dummy@example.com>Description: emulate base-filesEOFdpkg-deb --root-owner-group -b m18base m18base.debmkdir -p m18dummy/DEBIAN m18dummy/libcat >m18dummy/DEBIAN/control <<EOFPackage: m18dummy
Architecture: allVersion: 1Maintainer: Dummy <dummy@exmaple.com>Description: test packageEOFecho "m18dummy 1" >m18dummy/lib/m18dummydpkg-deb --root-owner-group -b m18dummy m18dummy_1.debmkdir m18dummy/usrmv m18dummy/lib m18dummy/usr/libecho "
m18dummy 2" >m18dummy/usr/lib/m18dummysed -i -e '/^Version:/s/1/2/' m18dummy/DEBIAN/controldpkg-deb --root-owner-group -b m18dummy m18dummy_2.deb# We are testing dpkg-divert in a non-root chroot located here.# This works as long as all maintainer
scripts honour DPKG_ROOT.INST_ROOT=rootsudo_dpkg() { PATH="/sbin:$PATH" dpkg --root "$INST_ROOT" --log "$INST_ROOT/var/log/dpkg.log" --force-not-root --force-script-chrootless "$@"}install_aliased() { sudo_dpkg -i m18dummy_1.deb}install_
canonical() { sudo_dpkg -i m18dummy_2.deb}remove_package() { sudo_dpkg -r m18dummy}setup_root() { rm -Rf "$INST_ROOT" mkdir -p "$INST_ROOT/var/log" sudo_dpkg -i m18base.deb}# Proposed diversion duplication mechanism## Q1: Should
duplicated diversions have aliased diversion targets?# A1: No! While dpkg is smart enough to not clobber undiverted files when# moving them from / to /usr, this safety does not apply when the file is# diverted. Instead such files reliably get lost.
Diversion targets must never# alias.## Q2: Can dpkg -S be used to determine whether --rename should be used?# A2: No! Before removing a diversion, dpkg -S will succeed just because# the file has been diverted regardless of whether some package
installed it.## Q3: Does it become any simpler if the diverted file is assumed canonical# before diversion removal?# A3: Yes! Pass --no-rename for aliased and --rename for canonical and see# far below. This requires that the diverting package
depends (with a version)# on the canonicalized divertee. It is not sufficient to declare Breaks with# the aliased version as removing it results in an inconsistency.## Q4: Why can I use --rename for diversion removal?# A4: Because only one of the
diversion targets exists (as a result of A1) and# the other rename will not have any effect.add_diversions() { if test "$(dpkg-divert --root "$INST_ROOT" --truename /usr/lib/m18dummy)" != /usr/lib/m18dummy; then : elif test "$(dpkg-divert --root "$
INST_ROOT" --truename /lib/m18dummy)" != /lib/m18dummy; then : elif dpkg --root "$INST_ROOT" -S /usr/lib/m18dummy >/dev/null 2>&1; then mv "$INST_ROOT/usr/lib/m18dummy" "$INST_ROOT/usr/lib/m18dummy.distrib" elif dpkg --root "$INST_ROOT" -S /lib/
m18dummy >/dev/null 2>&1; then mv "$INST_ROOT/lib/m18dummy" "$INST_ROOT/lib/m18dummy.distrib.usr-is-merged" elif test -e "$INST_ROOT/usr/lib/m18dummy"; then echo "attempting to dpkg-divert unowned file" exit 1 fi dpkg-divert --root "$INST_ROOT" -
-add --no-rename --divert /lib/m18dummy.distrib.usr-is-merged /lib/m18dummy dpkg-divert --root "$INST_ROOT" --add --no-rename /usr/lib/m18dummy echo diverted > "$INST_ROOT/usr/lib/m18dummy"}remove_diversions() { if test "$(dpkg-divert --root "$INST_
ROOT" --truename /usr/lib/m18dummy)" != /usr/lib/m18dummy || test "$(dpkg-divert --root "$INST_ROOT" --truename /lib/m18dummy)" != /lib/m18dummy; then rm "$INST_ROOT/usr/lib/m18dummy" fi dpkg-divert --root "$INST_ROOT" --remove --rename /usr/lib/
m18dummy dpkg-divert --root "$INST_ROOT" --remove --rename /lib/m18dummy}test_fail() { echo "$*" 1>&2 ls -la "$INST_ROOT/lib/" dpkg-divert --root "$INST_ROOT" --list dpkg --root "$INST_ROOT" --verify exit 1}# Assert that the file "$INST_ROOT$
1" contains "$2".assert_file_contains() { if ! test -e "$INST_ROOT$1"; then test_fail "expected $1 to contain $2, but it does not exist" fi if ! cmp - "$INST_ROOT$1" <<EOF$2EOF then test_fail "FAIL.expected $1 to contain $2, but it conatsin $(
cat "$1")" fi}# Assert that "$INST_ROOT$1" does not exist.assert_file_absent() { if test -e "$INST_ROOT$1"; then test_fail "FAIL. expected $1 to be absent, but it exists" fi}# Assert that all is well:# * dpkg's database matches reality.# *
Files contain what we expect them to contain.# * No left-over files from earlier diversions.validate_common() { bad="$(dpkg --root "$INST_ROOT" --verify)" if test -n "$bad"; then test_fail "dpkg --verify has non-empty output $bad" fi test -h "$
INST_ROOT/lib" || test_fail "/lib vanished" bad="$(find "$INST_ROOT" ! \( \ \( -path "$INST_ROOT/var" -prune \) -o \ -path "$INST_ROOT" -o \ -path "$INST_ROOT/lib" -o \ -path "$INST_ROOT/usr" -o \ -path "$INST_ROOT/usr/lib" -o \ -path "$INST_
ROOT/usr/lib/m18dummy" -o \ -path "$INST_ROOT/usr/lib/m18dummy.distrib" -o \ -path "$INST_ROOT/usr/lib/m18dummy.distrib.usr-is-merged" \) )" if test -n "$bad"; then test_fail "unexpected files $bad" fi}validate_absent() { validate_common
assert_file_absent /usr/lib/m18dummy assert_file_absent /usr/lib/m18dummy.distrib assert_file_absent /usr/lib/m18dummy.distrib.usr-is-merged}validate_absent_diverted() { validate_common assert_file_contains /usr/lib/m18dummy "diverted" assert_file_
absent /usr/lib/m18dummy.distrib assert_file_absent /usr/lib/m18dummy.distrib.usr-is-merged}validate_aliased() { validate_common assert_file_contains /usr/lib/m18dummy "m18dummy 1" assert_file_absent /usr/lib/m18dummy.distrib assert_file_absent /
usr/lib/m18dummy.distrib.usr-is-merged}validate_aliased_diverted() { validate_common assert_file_contains /usr/lib/m18dummy "diverted" assert_file_absent /usr/lib/m18dummy.distrib assert_file_contains /usr/lib/m18dummy.distrib.usr-is-merged "
m18dummy 1"}validate_canonical() { validate_common assert_file_contains /usr/lib/m18dummy "m18dummy 2" assert_file_absent /usr/lib/m18dummy.distrib assert_file_absent /usr/lib/m18dummy.distrib.usr-is-merged}validate_canonical_diverted() {
validate_common assert_file_contains /usr/lib/m18dummy "diverted" assert_file_contains /usr/lib/m18dummy.distrib "m18dummy 2" assert_file_absent /usr/lib/m18dummy.distrib.usr-is-merged}testcase_absent() { validate_absent add_diversions validate_
absent_diverted remove_diversions validate_absent}testcase_install_aliased() { validate_absent add_diversions validate_absent_diverted install_aliased validate_aliased_diverted remove_diversions validate_aliased}testcase_installed_aliased()
{ validate_absent install_aliased validate_aliased add_diversions validate_aliased_diverted remove_diversions validate_aliased}testcase_install_canonical() { validate_absent add_diversions validate_absent_diverted install_canonical
validate_canonical_diverted remove_diversions validate_canonical}testcase_installed_canonical() { validate_absent install_canonical validate_canonical add_diversions validate_canonical_diverted remove_diversions validate_canonical}testcase_
upgrade_aliased() { validate_absent install_aliased validate_aliased add_diversions validate_aliased_diverted install_aliased validate_aliased_diverted remove_diversions validate_aliased}testcase_upgrade_canonical() { validate_absent
install_canonical validate_canonical add_diversions validate_canonical_diverted install_canonical validate_canonical_diverted remove_diversions validate_canonical}testcase_upgrade_move() { validate_absent install_aliased validate_aliased
add_diversions validate_aliased_diverted install_canonical validate_canonical_diverted remove_diversions validate_canonical}testcase_remove_aliased() { validate_absent install_aliased validate_aliased add_diversions validate_aliased_diverted
remove_package validate_absent_diverted remove_diversions validate_absent}testcase_remove_canonical() { validate_absent install_canonical validate_canonical add_diversions validate_canonical_diverted remove_package validate_absent_diverted
remove_diversions validate_absent}testcase_absent_diverted_idempotent() { validate_absent add_diversions validate_absent_diverted add_diversions validate_absent_diverted remove_diversions validate_absent remove_diversions validate_absent}
testcase_aliased_diverted_idempotent() { validate_absent install_aliased validate_aliased add_diversions validate_aliased_diverted add_diversions validate_aliased_diverted remove_diversions validate_aliased remove_diversions validate_aliased}
testcase_canonical_diverted_idempotent() { validate_absent install_canonical validate_canonical add_diversions validate_canonical_diverted add_diversions validate_canonical_diverted remove_diversions validate_canonical remove_diversions
validate_canonical}for testcase in testcase_absent testcase_install_aliased testcase_installed_aliased testcase_install_canonical testcase_installed_canonical testcase_upgrade_aliased testcase_upgrade_canonical testcase_upgrade_move testcase_remove_
aliased testcase_remove_canonical testcase_absent_diverted_idempotent testcase_aliased_diverted_idempotent testcase_canonical_diverted_idempotent; do echo "$testcase" setup_root "$testcase" echo OKdone# Simplified implementation when assuming that
the diverted file becomes# canonical quickly after creating the diversion.add_diversions() { dpkg-divert --root "$INST_ROOT" --add --no-rename --divert /lib/m18dummy.distrib.usr-is-merged /lib/m18dummy dpkg-divert --root "$INST_ROOT" --add --rename /
usr/lib/m18dummy echo diverted > "$INST_ROOT/usr/lib/m18dummy"}remove_diversions() { if test "$(dpkg-divert --root "$INST_ROOT" --truename /usr/lib/m18dummy)" != /usr/lib/m18dummy || test "$(dpkg-divert --root "$INST_ROOT" --truename /lib/m18dummy)
" != /lib/m18dummy; then rm "$INST_ROOT/usr/lib/m18dummy" fi dpkg-divert --root "$INST_ROOT" --remove --rename /usr/lib/m18dummy dpkg-divert --root "$INST_ROOT" --remove --no-rename /lib/m18dummy}testcase_upgrade_move_relaxed() { validate_absent
install_aliased validate_aliased add_diversions # temporarily does not validate, dependency of diverter on divertee required. install_canonical validate_canonical_diverted remove_diversions validate_canonical}for testcase in testcase_absent
testcase_install_canonical testcase_installed_canonical testcase_upgrade_canonical testcase_upgrade_move_relaxed testcase_remove_canonical testcase_absent_diverted_idempotent testcase_canonical_diverted_idempotent; do echo "$testcase" setup_root "$
testcase" echo OKdonefor testcase in testcase_installed_aliased testcase_installed_aliased testcase_upgrade_aliased testcase_upgrade_move testcase_remove_aliased testcase_aliased_diverted_idempotent; do echo "expecting failure from $testcase" setup_
root if ( "$testcase"; ); then echo "unexpected success from $testcase" exit 1 fi echo OKdone

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Branden,

On Thu, Jan 09, 2025 at 01:41:16PM -0600, G. Branden Robinson wrote:

Fundamentally, I assumed that dpkg-divert --rename would only move a
file if it was considered installed according to the dpkg database.
What really happens is that --rename moves a file as long as it
exists.

Is that within dpkg's specification? If it is, in fact, a bug, should
we consider tackling it for the next development cycle?

Really? We are very deep in the land of implementation-defined
behaviour. From a specification point of view, I'd call doing diversions
on aliased files undefined behaviour. It really does not matter at all
whether we call this a bug or anything. At the time our maintainer
scripts run, we must expect them to run with bookworm's or trixie's dpkg
and we cannot rely on any change in behaviour to be applied to the
former one. The only thing that we care about is that as little as
possible implementation-defined behaviour in this area changes between
bookworm and trixie. Luckily, Guillem put his effort into other valuable
areas of dpkg that are less of a concern for the /usr-move.

Then when it comes to the next development cycle, the idea is that we
have left aliasing behind. In the common case, if there is an untracked
file in a location that is being diverted, the present --rename
semantics arguably are useful as the untracked file will be renamed (on installation) and back (on removal). I asked Guillem off list to
consider documenting this aspect.

Helmut

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Jan 08, 2025 at 03:21:42PM +0100, Helmut Grohne wrote:

Since I was failing to get this right earlier, I figured I could as well write some tests. As a result, I'm attaching a shell script. It
constructs a few demo packages and attempts a hopefully exhaustive combination of using them and dpkg-divert by writing its experiments to
the working directory. Running it should be fairly boring. If you happen
to review it, I appreciate a response.

Despite multiple requests to actually attach my tests, nobody seems to
have read them and spot the missing test cases. And of course they were failing.

The case of upgrading from a single, aliased diversion to a duplicated diversion was not covered. In this case, we want the canonicalized
diversion target to be used by the canonicalized diversion and
therefore, the aliased diversion must change its diversion target.
Failure to do so results in file loss and/or left-over files.

I'm attaching an updated version that tests the transition from aliased diversion to duplicated diversion both in the case of the divertee being
absent and present. It's a lot of branches now, but I don't quite see
how we could do it with less.

Again, I appreciate someone actually reading the test cases and telling
me where I am wrong such that I don't have to redo too many redone
patches. At least the one I submitted for bfh-container fails an
additional test case.

Helmut

#!/bin/sh# shellcheck disable=SC2317 # Our way of running test cases confuses shellcheck.set -e# Create three Debian packages:# * m18base.deb emulates base-files and installs /lib and /usr/lib.# * m18dummy_1.deb installs /lib/m18dummy.# *
m18dummy_2.deb installs /usr/lib/m18dummy.rm -Rf root m18base m18dummyrm -f m18base.deb m18dummy_1.deb m18dummy_2.debmkdir -p m18base/DEBIAN m18base/usr/libln -s usr/lib m18base/libcat >m18base/DEBIAN/control <<EOFPackage: m18baseArchitecture:
allEssential: yesVersion: 1Maintainer: Dummy <dummy@example.com>Description: emulate base-filesEOFdpkg-deb --root-owner-group -b m18base m18base.debmkdir -p m18dummy/DEBIAN m18dummy/libcat >m18dummy/DEBIAN/control <<EOFPackage: m18dummy
Architecture: allVersion: 1Maintainer: Dummy <dummy@exmaple.com>Description: test packageEOFecho "m18dummy 1" >m18dummy/lib/m18dummydpkg-deb --root-owner-group -b m18dummy m18dummy_1.debmkdir m18dummy/usrmv m18dummy/lib m18dummy/usr/libecho "
m18dummy 2" >m18dummy/usr/lib/m18dummysed -i -e '/^Version:/s/1/2/' m18dummy/DEBIAN/controldpkg-deb --root-owner-group -b m18dummy m18dummy_2.deb# We are testing dpkg-divert in a non-root chroot located here.# This works as long as all maintainer
scripts honour DPKG_ROOT.INST_ROOT=rootsudo_dpkg() { PATH="/sbin:$PATH" dpkg --root "$INST_ROOT" --log "$INST_ROOT/var/log/dpkg.log" --force-not-root --force-script-chrootless "$@"}install_aliased() { sudo_dpkg -i m18dummy_1.deb}install_
canonical() { sudo_dpkg -i m18dummy_2.deb}remove_package() { sudo_dpkg -r m18dummy}setup_root() { rm -Rf "$INST_ROOT" mkdir -p "$INST_ROOT/var/log" sudo_dpkg -i m18base.deb}# Proposed diversion duplication mechanism## Q1: Should
duplicated diversions have aliased diversion targets?# A1: No! While dpkg is smart enough to not clobber undiverted files when# moving them from / to /usr, this safety does not apply when the file is# diverted. Instead such files reliably get lost.
Diversion targets must never# alias.## Q2: Can dpkg -S be used to determine whether --rename should be used?# A2: No! Before removing a diversion, dpkg -S will succeed just because# the file has been diverted regardless of whether some package
installed it.## Q3: Does it become any simpler if the diverted file is assumed canonical# before diversion removal?# A3: Yes! Pass --no-rename for aliased and --rename for canonical and see# far below. This requires that the diverting package
depends (with a version)# on the canonicalized divertee. It is not sufficient to declare Breaks with# the aliased version as removing it results in an inconsistency.## Q4: Why can I use --rename for diversion removal?# A4: Because only one of the
diversion targets exists (as a result of A1) and# the other rename will not have any effect.add_old_diversion() { # A simple diversion of the aliased path only as was done pre-trixie. dpkg-divert --root "$INST_ROOT" --add --rename /lib/m18dummy
echo "diverted old" > "$INST_ROOT/usr/lib/m18dummy"}add_diversions() { if test "$(dpkg-divert --root "$INST_ROOT" --truename /usr/lib/m18dummy)" != /usr/lib/m18dummy; then # Canonically diverted already. Not renaming. : elif test "$(dpkg-divert -
-root "$INST_ROOT" --truename /lib/m18dummy)" = /lib/m18dummy.distrib; then # Aliased diversion with earlier target. Rename target. dpkg-divert --root "$INST_ROOT" --remove --no-rename /lib/m18dummy if test -e "$INST_ROOT/lib/m18dummy.distrib" -o -
h "$INST_ROOT/lib/m18dummy.distrib"; then mv "$INST_ROOT/lib/m18dummy.distrib" "$INST_ROOT/lib/m18dummy.distrib.usr-is-merged" fi elif test "$(dpkg-divert --root "$INST_ROOT" --truename /lib/m18dummy)" != /lib/m18dummy; then # Aliased diversion.
Not renaming. : elif dpkg --root "$INST_ROOT" -S /usr/lib/m18dummy >/dev/null 2>&1; then # Not diverted. Installed as canonical. Rename. mv "$INST_ROOT/usr/lib/m18dummy" "$INST_ROOT/usr/lib/m18dummy.distrib" elif dpkg --root "$INST_ROOT" -S /lib/
m18dummy >/dev/null 2>&1; then # Not diverted. Installed as aliased. Rename. mv "$INST_ROOT/lib/m18dummy" "$INST_ROOT/lib/m18dummy.distrib.usr-is-merged" elif test -e "$INST_ROOT/usr/lib/m18dummy" -o -h "$INST_ROOT/usr/lib/m18dummy"; then # Not
diverted nor installed, but present. echo "attempting to dpkg-divert unowned file" exit 1 fi dpkg-divert --root "$INST_ROOT" --add --no-rename --divert /lib/m18dummy.distrib.usr-is-merged /lib/m18dummy dpkg-divert --root "$INST_ROOT" --add --no-
rename /usr/lib/m18dummy echo diverted > "$INST_ROOT/usr/lib/m18dummy"}remove_diversions() { if test "$(dpkg-divert --root "$INST_ROOT" --truename /usr/lib/m18dummy)" != /usr/lib/m18dummy || test "$(dpkg-divert --root "$INST_ROOT" --truename /lib/
m18dummy)" != /lib/m18dummy; then rm "$INST_ROOT/usr/lib/m18dummy" fi # Only one of the diversion targets should exist. Hence we may rename both. dpkg-divert --root "$INST_ROOT" --remove --rename /usr/lib/m18dummy dpkg-divert --root "$INST_ROOT" --
remove --rename /lib/m18dummy}test_fail() { echo "$*" 1>&2 ls -la "$INST_ROOT/lib/" dpkg-divert --root "$INST_ROOT" --list dpkg --root "$INST_ROOT" --verify exit 1}# Assert that the file "$INST_ROOT$1" contains "$2".assert_file_contains() {
if ! test -e "$INST_ROOT$1"; then test_fail "expected $1 to contain $2, but it does not exist" fi if ! cmp - "$INST_ROOT$1" <<EOF$2EOF then test_fail "FAIL.expected $1 to contain $2, but it conatsin $(cat "$1")" fi}# Assert that "$INST_ROOT$
1" does not exist.assert_file_absent() { if test -e "$INST_ROOT$1"; then test_fail "FAIL. expected $1 to be absent, but it exists" fi}# Assert that all is well:# * dpkg's database matches reality.# * Files contain what we expect them to
contain.# * No left-over files from earlier diversions.validate_common() { bad="$(dpkg --root "$INST_ROOT" --verify)" if test -n "$bad"; then test_fail "dpkg --verify has non-empty output $bad" fi test -h "$INST_ROOT/lib" || test_fail "/lib
vanished" bad="$(find "$INST_ROOT" ! \( \ \( -path "$INST_ROOT/var" -prune \) -o \ -path "$INST_ROOT" -o \ -path "$INST_ROOT/lib" -o \ -path "$INST_ROOT/usr" -o \ -path "$INST_ROOT/usr/lib" -o \ -path "$INST_ROOT/usr/lib/m18dummy" -o \ -
path "$INST_ROOT/usr/lib/m18dummy.distrib" -o \ -path "$INST_ROOT/usr/lib/m18dummy.distrib.usr-is-merged" \) )" if test -n "$bad"; then test_fail "unexpected files $bad" fi}validate_absent() { validate_common assert_file_absent /usr/lib/
m18dummy assert_file_absent /usr/lib/m18dummy.distrib assert_file_absent /usr/lib/m18dummy.distrib.usr-is-merged}validate_absent_diverted() { validate_common assert_file_contains /usr/lib/m18dummy "diverted" assert_file_absent /usr/lib/m18dummy.
distrib assert_file_absent /usr/lib/m18dummy.distrib.usr-is-merged}validate_absent_diverted_old() { validate_common assert_file_contains /usr/lib/m18dummy "diverted old" assert_file_absent /usr/lib/m18dummy.distrib assert_file_absent /usr/lib/
m18dummy.distrib.usr-is-merged}validate_aliased() { validate_common assert_file_contains /usr/lib/m18dummy "m18dummy 1" assert_file_absent /usr/lib/m18dummy.distrib assert_file_absent /usr/lib/m18dummy.distrib.usr-is-merged}validate_aliased_
diverted() { validate_common assert_file_contains /usr/lib/m18dummy "diverted" assert_file_absent /usr/lib/m18dummy.distrib assert_file_contains /usr/lib/m18dummy.distrib.usr-is-merged "m18dummy 1"}validate_aliased_diverted_old() { validate_common
assert_file_contains /usr/lib/m18dummy "diverted old" assert_file_contains /usr/lib/m18dummy.distrib "m18dummy 1" assert_file_absent /usr/lib/m18dummy.distrib.usr-is-merged}validate_canonical() { validate_common assert_file_contains /usr/lib/
m18dummy "m18dummy 2" assert_file_absent /usr/lib/m18dummy.distrib assert_file_absent /usr/lib/m18dummy.distrib.usr-is-merged}validate_canonical_diverted() { validate_common assert_file_contains /usr/lib/m18dummy "diverted" assert_file_contains /
usr/lib/m18dummy.distrib "m18dummy 2" assert_file_absent /usr/lib/m18dummy.distrib.usr-is-merged}testcase_absent() { validate_absent add_diversions validate_absent_diverted remove_diversions validate_absent}testcase_absent_duplicate() {
validate_absent add_old_diversion validate_absent_diverted_old add_diversions validate_absent_diverted}testcase_install_aliased() { validate_absent add_diversions validate_absent_diverted install_aliased validate_aliased_diverted remove_
diversions validate_aliased}testcase_installed_aliased() { validate_absent install_aliased validate_aliased add_diversions validate_aliased_diverted remove_diversions validate_aliased}testcase_installed_aliased_duplicate() { validate_
absent install_aliased validate_aliased add_old_diversion validate_aliased_diverted_old add_diversions validate_aliased_diverted}testcase_install_canonical() { validate_absent add_diversions validate_absent_diverted install_canonical
validate_canonical_diverted remove_diversions validate_canonical}testcase_installed_canonical() { validate_absent install_canonical validate_canonical add_diversions validate_canonical_diverted remove_diversions validate_canonical}testcase_
upgrade_aliased() { validate_absent install_aliased validate_aliased add_diversions validate_aliased_diverted install_aliased validate_aliased_diverted remove_diversions validate_aliased}testcase_upgrade_canonical() { validate_absent
install_canonical validate_canonical add_diversions validate_canonical_diverted install_canonical validate_canonical_diverted remove_diversions validate_canonical}testcase_upgrade_move() { validate_absent install_aliased validate_aliased
add_diversions validate_aliased_diverted install_canonical validate_canonical_diverted remove_diversions validate_canonical}testcase_remove_aliased() { validate_absent install_aliased validate_aliased add_diversions validate_aliased_diverted
remove_package validate_absent_diverted remove_diversions validate_absent}testcase_remove_canonical() { validate_absent install_canonical validate_canonical add_diversions validate_canonical_diverted remove_package validate_absent_diverted
remove_diversions validate_absent}testcase_absent_diverted_idempotent() { validate_absent add_diversions validate_absent_diverted add_diversions validate_absent_diverted remove_diversions validate_absent remove_diversions validate_absent}
testcase_aliased_diverted_idempotent() { validate_absent install_aliased validate_aliased add_diversions validate_aliased_diverted add_diversions validate_aliased_diverted remove_diversions validate_aliased remove_diversions validate_aliased}
testcase_canonical_diverted_idempotent() { validate_absent install_canonical validate_canonical add_diversions validate_canonical_diverted add_diversions validate_canonical_diverted remove_diversions validate_canonical remove_diversions
validate_canonical}for testcase in testcase_absent testcase_absent_duplicate testcase_install_aliased testcase_installed_aliased testcase_installed_aliased_duplicate testcase_install_canonical testcase_installed_canonical testcase_upgrade_aliased
testcase_upgrade_canonical testcase_upgrade_move testcase_remove_aliased testcase_remove_canonical testcase_absent_diverted_idempotent testcase_aliased_diverted_idempotent testcase_canonical_diverted_idempotent; do echo "$testcase" setup_root "$
testcase" echo OKdone# Simplified implementation when assuming that the diverted file becomes# canonical quickly after creating the diversion.add_diversions() { dpkg-divert --root "$INST_ROOT" --add --no-rename --divert /lib/m18dummy.distrib.usr-
is-merged /lib/m18dummy dpkg-divert --root "$INST_ROOT" --add --rename /usr/lib/m18dummy echo diverted > "$INST_ROOT/usr/lib/m18dummy"}remove_diversions() { if test "$(dpkg-divert --root "$INST_ROOT" --truename /usr/lib/m18dummy)" != /usr/lib/
m18dummy || test "$(dpkg-divert --root "$INST_ROOT" --truename /lib/m18dummy)" != /lib/m18dummy; then rm "$INST_ROOT/usr/lib/m18dummy" fi dpkg-divert --root "$INST_ROOT" --remove --rename /usr/lib/m18dummy dpkg-divert --root "$INST_ROOT" --remove -
-no-rename /lib/m18dummy}testcase_upgrade_move_relaxed() { validate_absent install_aliased validate_aliased add_diversions # temporarily does not validate, dependency of diverter on divertee required. install_canonical validate_canonical_
diverted remove_diversions validate_canonical}for testcase in testcase_absent testcase_install_canonical testcase_installed_canonical testcase_upgrade_canonical testcase_upgrade_move_relaxed testcase_remove_canonical testcase_absent_diverted_
idempotent testcase_canonical_diverted_idempotent; do echo "$testcase" setup_root "$testcase" echo OKdonefor testcase in testcase_absent_duplicate testcase_installed_aliased testcase_installed_aliased testcase_installed_aliased_duplicate testcase_
upgrade_aliased testcase_upgrade_move testcase_remove_aliased testcase_aliased_diverted_idempotent; do echo "expecting failure from $testcase" setup_root if ( "$testcase"; ); then echo "unexpected success from $testcase" exit 1 fi echo OKdone

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Jan 08, 2025 at 03:21:43PM +0100, Helmut Grohne wrote:
codesearching DEP17.*M18 (and a few other patterns) suggest the

following packages needing an update:
* bfh-metapackages

Open #1092955. I first got it wrong and then sent another update.

* clonezilla

Open #1093135. Though this one was broken in more ways and I guess it is
dead code.

* cryptsetup-nuke-password (and cryptsetup)

Not affected in a non-obvious way. cryptsetup-nuke-password performs the duplicate dpkg-divert --add --rename. It first renames the canonicalized
path wrongly renames an aliased askpass. However, it also depends on the canonicalized cryptsetup so by the time its postinst runs, the wrongly
moved file is overwritten by the cryptsetup update and things are fine
again.

* debian-installer-utils

Open https://salsa.debian.org/installer-team/debian-installer-utils/-/merge_requests/11

* isc-dhcp

Not affected in a non-obvious way. It does the duplicate dpkg-divert
--add --rename and therefore moves an aliased file to the wrong location
in its preinst. However, the diverting package also depends on a version
of the divertee that is canonicalized. The upgrade of the divertee will
clobber the wrongly moved file and after postinst the effects are ok.
This kinda is the simplified case in m18_diversions.sh.

* molly-guard

Open. #1093132. Avoiding --rename as much as possible now.

* live-build

Open https://salsa.debian.org/live-team/live-build/-/merge_requests/394#note_569976

* open-infrastructure-system-tools

Open. #1093131. Similar to debian-installer-utils.

* opensysusers

The duplicate --rename is there, but in the mean time systemd-sysusers implementations started conflicting with one another, so the use of
diversions is no longer required. I propose dropping the diversion mess entirely: #1092745.

* piuparts

Fixed. Aparently, I understood this back in March 2024 and then failed
to transfer what I learned to other packages. :-( https://salsa.debian.org/debian/piuparts/-/commit/1b9a3843964b42b983921a1b011f14d61e430944

* progress-linux-metapackages

Open #1092956. Similar to bfh-container.

* systemd

Not affected. It is being diverted, but does no --rename.

* zutils (and gzip)

Installing zutils/trixie on bookworm, looses gzip files. #1092737 fixed
in zutils/1.14-1.

* runit

Not affected. It is being diverted, but does no --rename.

* sysvinit

Not affected. It is being diverted and it diverts manual pages with
--rename.

So this hopefully concludes the mess as I am not aware of any other
affected cases. What this does not handle is the breakage left behind
trixie or sid systemd that have taken all intermediate steps. If you
find that you have any files named *.usr-is-merged, that's a bad sign
and you may reach out to me. My focus has been fixing the bookworm ->
trixie upgrade, but we may also handle unstable upgrades if that happens
to affect non-trivial amounts of users.

Helmut

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)