To: debian-devel@lists.debian.org (Debian developers list)

This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------siMqeZFXqEzEUqYv5s4FMaCg
Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64

SGkgSnVsaWFuLA0KDQpPbiAyOC0wNC0yMDIyIDA5OjMzLCBKdWxpYW4gR2lsYmV5IHdyb3Rl Og0KPiBJdCB3b3VsZCBiZSByZWFsbHkgdXNlZnVsIHRvIGJlIGFibGUgdG8gc2V0IHVwIG15 IGxvY2FsIHNidWlsZA0KPiBlbnZpcm9ubWVudCBpbiB0aGUgc2FtZSB3YXkgYXMgdGhlIERl YmlhbiBtYWNoaW5lcyAoYnVpbGRkIGFuZA0KPiBjaS5kZWJpYW4ubmV0KSBmb3IgdGVzdGlu ZyBwdXJwb3Nlcy4NCg0KQXMgSSd2ZSBuZXZlciB1c2VkIHNidWlsZCBteXNlbGYsIEkgY2Fu J3QgdGVsbCB5b3UgaG93IHRvIHNldCBpdCB1cC4gQnV0IA0Kb24gY2kuZGViaWFuLm5ldCwg d2UgY3VycmVudGx5IGhhdmUgdGhlIGx4YyBiYWNrZW5kcyB3aGljaCBhcmUgc2V0dXAgDQp3 aXRoIGF1dG9wa2d0ZXN0LWJ1aWxkLWx4Yy4gRGlmZmVyZW50IGJhY2tlbmRzIG1heSBiZWhh dmUgc2xpZ2h0bHkgDQpkaWZmZXJlbnQgYW5kIHdlIHN0aWxsIGhhdmUgdGhlIGFtYml0aW9u IChidXQgaXQgd2lsbCBub3QgaGFwcGVuIGFueSANCnRpbWUgc29vbiBJIGZlYXIpIHRvIHN1 cHBvcnQgaXNvbGF0aW9uLW1hY2hpbmUgd2hpY2ggd2lsbCBiZSBzZXR1cCB3aXRoIA0KYSBk aWZmZXJlbnQgdG9vbC4NCg0KSSdtIGFic29sdXRlbHkgbm90IGNvbnZpbmNlZCB0aGF0IG9u IHRoZSBidWlsZGRzIGFuZCBvbiBjaS5kZWJpYW4ubmV0IA0KdGhlIGVudmlyb25tZW50IGlz IHNldHVwICJpbiB0aGUgc2FtZSB3YXkiIHNvIG1heWJlIHlvdXIgY2hhbGxlbmdlIGhhcyAN Cm5vIHBvc3NpYmxlIG91dGNvbWUuDQoNClBhdWwNCg==

--------------siMqeZFXqEzEUqYv5s4FMaCg--

-----BEGIN PGP SIGNATURE-----

wsB5BAABCAAjFiEEWLZtSHNr6TsFLeZynFyZ6wW9dQoFAmJqSjAFAwAAAAAACgkQnFyZ6wW9dQqX 4ggAreb4zNWH/PEsWLKnEf7Z72E1F/OSUPt5TlGdwMEvrttAcDxFnnxeEDF9bd9DDF8xTgkj+3sR hfEP1LB1xUU8s1prqRfaQeS4kp1k7wRyBKsb/LNF/dpYv4NQjcSCrlhR5N32ius6C7P8di4p8vEm prx6UOdTSHP4hNHlNFSJsUQu3fFCqAeGvy2EHGBKBIYzfoUY0QfVxhoeh6lYS1poGIjiiEpmdVkt oUTYvvHrb6ytvZrLztRfr6dIpPqdhU0EnPE161dqoLN1EH0Saq5IDQWfBCQ+NNxptT314p4mZz3U j2DAdLJE4evK+jgI2kcmH567OttiomZQWR7e9LVjeA==
=CHGm
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thu, 28 Apr 2022 at 08:33:02 +0100, Julian Gilbey wrote:

When I run autopkgtest using sbuild on my own machine, I sometimes get warnings or breakages because certain environment variables are either
unset or have "broken" values, whereas I see that on ci.debian.net and
the buildds, the builds run without problem.

What autopkgtest backend are you using? (autopkgtest-virt-schroot? autopkgtest-virt-qemu? ...) There are several, and their behaviour can be
very different, depending on the level of isolation they provide.

Having a concrete example of the command-line and configuration you use,
and a package that has warnings or breakage in this configuration, would
be useful.

It would be really useful to be able to set up my local sbuild
environment in the same way as the Debian machines (buildd and
ci.debian.net) for testing purposes.

buildds do not run autopkgtest, only sbuild, so they run build-time tests (dh_auto_test or similar) but do not interact with debian/tests/.

ci.debian.net currently uses autopkgtest-virt-lxc via the debci package.

HOME (pointing to where?)

sbuild
------

There is no guarantee about $HOME while a Debian package is being built.
Policy §4.9 [0] says it is a bug for the build process of a package to
write into the real $HOME.

[0] https://www.debian.org/doc/debian-policy/ch-source.html#main-building-script-debian-rules

In sbuild, $HOME is set to the home directory of the user running the
build, but that directory is not guaranteed to exist inside the chroot.
Most official buildds set it to /nonexistent, specifically so that builds
that rely on $HOME will fail.

debhelper compat levels >= 13 set a temporary $HOME for the duration
of the build, to make it easier for packages to comply with the Policy
§4.9 requirement.

autopkgtest
-----------

According to the autopkgtest spec[1],

Tests can expect that the $HOME environment variable to be set to
a directory that exists and is writeable by the user running the test.

Depending on backend, that directory might be your real home directory,
or it might be a transient/expendable home directory that it is OK for
the test to damage. It would probably be good to have a restriction[2]
that has the effect of guaranteeing that an expendable home directory
is used, as it is in autopkgtest-virt-qemu.

[1] https://salsa.debian.org/ci-team/autopkgtest/blob/HEAD/doc/README.package-tests.rst
[2] https://salsa.debian.org/ci-team/autopkgtest/blob/HEAD/doc/README.package-tests.rst#defined-restrictions

LC_ALL (or something similar; what is its setting?)

LC_ALL is not normally set on "real" Debian systems. Usually, only $LANG
and $LANGUAGE are set, even on a full desktop system (for example my GNOME desktop has LANG=en_GB.utf8 and LANGUAGE=en_GB:en, but no LC_ALL).

LC_ALL is intended to be the "big hammer" that overrides everything else.

sbuild
------

There are no guarantees about the locale with which a Debian package
will be built. The reproducible-builds infrastructure regularly builds
packages with non-English locales, which in practice often causes test
failures or non-reproducibility.

debhelper sets LC_ALL=C.UTF-8 for Meson/Ninja packages. debian/rules can
have `export LC_ALL=C.UTF-8`, if you want to (I would recommend this).

Even if you set the locale explicitly, locales other than C (aka
POSIX) and C.UTF-8 (aka C.utf8) are not guaranteed to be available
inside the test environment, unless your test has a dependency on
locales-all. Several source packages (for example gtk4) have a script debian/tests/run-with-locales which generates additional locales
on-demand.

autopkgtest
-----------

The autopkgtest spec does not specify whether these variables will be set,
and if they are set, what their values will be.

Test scripts can set LC_ALL=C or LC_ALL=C.UTF-8 or individual locale environment variables, if you want them to.

Even if you set the locale explicitly, locales other than C and C.UTF-8
are not guaranteed to be available inside the test environment, unless
your test has a dependency on locales-all. Again,
debian/tests/run-with-locales from the gtk4 package might be useful.

and XDG_RUNTIME_DIR

sbuild
------

There is no guarantee about $XDG_RUNTIME_DIR while a Debian package is
being built. Policy §4.9 [0] indirectly implies that it is a bug for
the build process of a package to write into the real $XDG_RUNTIME_DIR.

sbuild does whatever schroot does. If I remember correctly, schroot
currently inherits XDG_RUNTIME_DIR from the host system, without doing
the setup to make it *work* inside the chroot. If true, that's a bug; but schroot is orphaned in Debian and seems mostly dead upstream (its upstream maintainer has left Debian, the most recent upstream release was in 2014,
and there was no response to a merge request that I sent a while ago) so
it is a bug that seems unlikely to be fixed unless someone in Debian forks schroot. I personally think making sbuild default to --chroot-mode=unshare
or --chroot-mode=autopkgtest, and using one of those on official buildds,
would be a better way forward than forking and fixing schroot.

debhelper compat levels >= 13 set a temporary $XDG_RUNTIME_DIR for the
duration of `dh_auto_test`, and unset it at all other times, to make it
easier for packages to comply with the Policy §4.9 requirement.

autopkgtest
-----------

The autopkgtest spec does not specify whether this will be set or not.

In practice, on "real" systems, XDG_RUNTIME_DIR is only available if libpam-systemd or libpam-elogind is installed and working. In practice
this means it will only be available in full-system containers, virtual machines or physical machines (Requirements: isolation-container or isolation-machine); only if they run a full init system like systemd-sysv
or sysvinit-core; and only if libpam-systemd or libpam-elogind is
installed and working (your test should have a Depends on libpam-systemd,
or similar, if you need this).

In simple chroot-based environments like schroot, or in non-full-system containers like Docker/Podman, the infrastructure to set a correct XDG_RUNTIME_DIR with its documented semantics[3] is not there, so ideally
it should be unset.

However, I think schroot currently inherits XDG_RUNTIME_DIR from the host system (see sbuild, above), so you might get a wrong XDG_RUNTIME_DIR
for tests whose Restrictions allow them to be run in a simple chroot.
Test scripts can work around this by setting it to a temporary directory.

smcv

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Thanks Paul and Simon for your very thoughtful and helpful replies!
Comments on excerpted text below.

On Thu, Apr 28, 2022 at 10:16:30AM +0100, Simon McVittie wrote:

On Thu, 28 Apr 2022 at 08:33:02 +0100, Julian Gilbey wrote:

When I run autopkgtest using sbuild on my own machine, I sometimes get warnings or breakages because certain environment variables are either unset or have "broken" values, whereas I see that on ci.debian.net and
the buildds, the builds run without problem.

What autopkgtest backend are you using? (autopkgtest-virt-schroot? autopkgtest-virt-qemu? ...) There are several, and their behaviour can be very different, depending on the level of isolation they provide.

I had been using the schroot backend. I hadn't realised that schroot
is abandoned upstream, though it seems to work quite nicely. I
haven't looked at lxc or qemu, though they sound like interesting
alternatives. (Happy to receive recommendations for what would be
simplest for an individual developer's machine!)

Having a concrete example of the command-line and configuration you use,
and a package that has warnings or breakage in this configuration, would
be useful.

I was trying to apply a patch to pymca to fix bug#1010258; see version 5.7.1+dfsg-2 of the package.

When building the package with sbuild, using the command line:

gbp buildpackage --git-builder=sbuild -d unstable -A --source-only-changes

the tests during package build time were fine, but the autopkgtest
failed. I fixed the autopkgtest by specifying:

LC_ALL=C HOME="$AUTOPKGTEST_TMP" xvfb-run ...

This makes perfect sense given what you wrote later.

ci.debian.net currently uses autopkgtest-virt-lxc via the debci package.

Ah, that would explain at least some of the difference with what I am
seeing.

HOME (pointing to where?)

sbuild
------

There is no guarantee about $HOME while a Debian package is being built. Policy §4.9 [0] says it is a bug for the build process of a package to
write into the real $HOME.

It may well be that these autopkgtests don't write to home but attempt
to access it to read from it. But I don't know.

debhelper compat levels >= 13 set a temporary $HOME for the duration
of the build, to make it easier for packages to comply with the Policy
§4.9 requirement.

That would explain why the package build part worked smoothly.

autopkgtest
-----------

According to the autopkgtest spec[1],

Tests can expect that the $HOME environment variable to be set to
a directory that exists and is writeable by the user running the test.

That's interesting; I don't find any code that sets HOME in
autopkgtest. And I just found this bug report, which points out that
this doesn't actually happen: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=986665
Now I understand that better, it seems that changing the autopkgtest
spec to say something more like the following, if it's true:

The HOME environment variable is inherited from the calling
process. It is up to the container manager to ensure that this
directory exists and is writeable by the user running the test.

Depending on backend, that directory might be your real home directory,
or it might be a transient/expendable home directory that it is OK for
the test to damage. It would probably be good to have a restriction[2]
that has the effect of guaranteeing that an expendable home directory
is used, as it is in autopkgtest-virt-qemu.

That sounds like a really good idea!

LC_ALL (or something similar; what is its setting?)

LC_ALL is not normally set on "real" Debian systems. Usually, only $LANG
and $LANGUAGE are set, even on a full desktop system (for example my GNOME desktop has LANG=en_GB.utf8 and LANGUAGE=en_GB:en, but no LC_ALL).

LC_ALL is intended to be the "big hammer" that overrides everything else.

Yes, indeed. But for some reason, pymca tries to set the locale at
some point in the tests. Setting LC_ALL=C, though it is a big hammer, certainly helped! I haven't tried using a more restricted thing like
LANG.

and XDG_RUNTIME_DIR

[...]

sbuild does whatever schroot does. If I remember correctly, schroot
currently inherits XDG_RUNTIME_DIR from the host system, [...]

It doesn't appear to do so (at least not by default).

debhelper compat levels >= 13 set a temporary $XDG_RUNTIME_DIR for the duration of `dh_auto_test`, and unset it at all other times, to make it easier for packages to comply with the Policy §4.9 requirement.

Ah, that would explain why the tests work during build time but give
warning messages with autopkgtest! (Incidentally, given that
typically XDG_RUNTIME_DIR is in the ephemeral /run filesystem, writing
to it would not violate policy 4.9.) I wonder whether autopkgtest
could consider setting up a temporary directory for this purpose like dh_auto_test does, rather than using the heavy machinery of
libpam-systemd?


So at the end of it all, it seems that setting up a temporary home
directory for HOME and runtime directory for XDG_RUNTIME_DIR in
autopkgtests that rely on these is the right way to go.

Best wishes,

Julian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thu, 28 Apr 2022 at 14:02:46 +0100, Julian Gilbey wrote:

On Thu, Apr 28, 2022 at 10:16:30AM +0100, Simon McVittie wrote:

According to the autopkgtest spec[1],

Tests can expect that the $HOME environment variable to be set to
a directory that exists and is writeable by the user running the test.

That's interesting; I don't find any code that sets HOME in
autopkgtest.

The purpose of autopkgtest is to run the tests in a realistic environment
that resembles a real Debian desktop or server, but it doesn't know much
about how the testbed system is set up, so it relies on the virtualization backend or the testbed itself to set HOME appropriately. Ideally, it
should be logging into a fully-working Debian system (perhaps under qemu
or lxc, or even on real hardware via ssh) and running tests there.

As I said on #986665, this seems like either a bug in
autopkgtest-virt-schroot, or misconfiguration in how you and Laurent
are configuring it (the schroot profile that you're telling it to use).

Now I understand that better, it seems that changing the autopkgtest
spec to say something more like the following, if it's true:

The HOME environment variable is inherited from the calling
process. It is up to the container manager to ensure that this
directory exists and is writeable by the user running the test.

No, that would be wrong for the lxc, qemu and ssh backends, which all
rely on an ordinary user with a home directory existing inside the
testbed system, and don't inherit anything from the caller.

(Incidentally, given that
typically XDG_RUNTIME_DIR is in the ephemeral /run filesystem, writing
to it would not violate policy 4.9.)

Policy §4.9 says that "Required targets must not attempt to write outside
of the unpacked source package tree", then makes a few exceptions for
locations like /tmp. /run is not one of those locations, so writing to
/run is not allowed.

I wonder whether autopkgtest
could consider setting up a temporary directory for this purpose like dh_auto_test does, rather than using the heavy machinery of
libpam-systemd?

No, I don't think it should. The purpose of autopkgtest is to run the
tests in a realistic environment that resembles a real Debian desktop
or server. If real Debian systems are going to use libpam-systemd (which
they often do), then autopkgtest should make it possible to assert that
the overall system works, by connecting to a full-system container
(lxc, systemd-nspawn) or a virtual or real machine (qemu, ssh) and using
it as-is.

In particular, it would make sense to write an autopkgtest that looks
like this pseudocode:

Restrictions: isolation-container
Depends: dbus-user-session, gnome-keyring

connect to org.gnome.keyring via D-Bus session bus
send a command to org.gnome.keyring
assert success

as an integration test that simulates gnome-keyring being installed
on an ordinary desktop system, and asserts that it is started either
on-demand or as a side-effect of logging in. (Replace gnome-keyring
with any service of your choice - dconf, Tracker, Telepathy, AT-SPI, PulseAudio, Pipewire, anything like that.)

That's not going to work with a mock XDG_RUNTIME_DIR, because the D-Bus
session bus that was run by dbus-user-session is listening on a socket in
the "real" XDG_RUNTIME_DIR provided by libpam-systemd, not the mock XDG_RUNTIME_DIR that you propose autopkgtest should provide.

Making the test script set up a mock XDG_RUNTIME_DIR and a mock
session bus would be a less useful test, because that only proves that gnome-keyring can work if you set up a mock environment by hand, and says nothing about whether gnome-keyring genuinely works on a normal system.

Running gnome-keyring-daemon by hand would also be a less useful test,
because if its maintainer had broken the D-Bus/systemd integration
necessary to start it on real, production systems, then the test would
not detect that (it would still succeed).

smcv

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thu, 2022-04-28 at 14:32 +0100, Simon McVittie wrote:

Making the test script set up a mock XDG_RUNTIME_DIR and a mock
session bus would be a less useful test, because that only proves that gnome-keyring can work if you set up a mock environment by hand, and says nothing about whether gnome-keyring genuinely works on a normal system.

Does that mean that autopkgtests that directly or indirectly use the dbus-run-session tool to setup a temporary D-Bus session are buggy?

If so, there are lots of dbus-run-session uses in Debian autopkgtests, including within the autopkgtests for dbus itself and parts of GNOME.

https://codesearch.debian.net/search?q=dbus-run-session+path%3Adebian%2Ftests%2F&literal=0

I've made this mistake myself in the mpv-mpris autopkgtests, got the
tests included upstream as a build-time test (where using a mock D-Bus
sessions seems correct) and then used those tests in the Debian
autopkgtests, but without removing the D-Bus mocking.

The other thing is testing graphical tools; should that be done in a
mock X11/Wayland session (using xvfb-run), or a real desktop session?

--
bye,
pabs

https://wiki.debian.org/PaulWise

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEYQsotVz8/kXqG1Y7MRa6Xp/6aaMFAmJrUNkACgkQMRa6Xp/6 aaNbxQ//U7O1MmsQskcsfiKdX0mJSGx1f8lVBfM79Jb1qcWetH+q9t6ikRg7oZRa gv8SQAhbqSqa6M/W2m2xfLfUarhChvOCTdhyGEKKH3vUC8nhO8jpRFvecMnHKDxb 25z6MUp7b3vQSQeFScGtSpwDy6lmD6LhbmDN+Cy9y9+68TDV02I7p3CaO1Y6fgIN VAs791EEaZsRa8u4soqqB7Kn75hXvc9iBNM4zBtCLwAsP7ldMYf1hxn8VsP5+6lA xVdRHmR0Wl4nqasaXV5umBF/zNTcEHeCqW25fWKOU5Ly5d9YgXLyD4cTgz02HexF /KPEpNn1rnuVenl+gCbK4hXFoyyEpG7Wy0LWWBiyBnzqcGAPWjhX1lgvBYwaYwU3 pScNl3feBGiVOvCrY4/FJFMOMNcnOd2WZTeN3HoAHRZjqp4UVX6SlFqaXR7C6tdN RHdleJ0QL7ovXSnsdPKzZcuwNK5z5S1wFR3Klk0Eo5niKnHepHqaSlncNPE3RJHZ 0GeKvLFgQthKoIbHe2kdIcHc4+oCjgkmraRM4Yt4p86xc2+Fw31RFfViHR47JJ+9 vQryoBsD3CTJG07jBMeAPir6DeQuI4rF8jCxqS8xMxt6BlFl+qeTYS+eeWzvgukS HYwWiLF50JhSSyYyb0gYOrMy+5AIlLekHlaRuvUZbVdfoCsR4YY=
=16Bo
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thu, Apr 28, 2022 at 02:32:49PM +0100, Simon McVittie wrote:

That's interesting; I don't find any code that sets HOME in
autopkgtest.

The purpose of autopkgtest is to run the tests in a realistic environment that resembles a real Debian desktop or server, but it doesn't know much about how the testbed system is set up, so it relies on the virtualization backend or the testbed itself to set HOME appropriately. Ideally, it
should be logging into a fully-working Debian system (perhaps under qemu
or lxc, or even on real hardware via ssh) and running tests there.

As I said on #986665, this seems like either a bug in autopkgtest-virt-schroot, or misconfiguration in how you and Laurent
are configuring it (the schroot profile that you're telling it to use).

I've thought a little bit more about this, and I realise that the
devil is in the detail, in particular the phrase "a realistic
environment that resembles a real Debian desktop or server". A
realistic environment is likely to have lots of random packages and
services available. It would make some sense, therefore, to be more
explicit about what is expected in this environment by default.
Here's a very small list to begin with:

1. All essential packages are installed.
2. HOME points to a readable and writeable directory. [What is in
this directory? Should it be cleaned before every use?]
3. XDG_RUNTIME_DIR points to ... / maybe this should be:
libpam-systemd is installed?
4. Locale LC_ALL ... should be set up in a meaningful way.
5. ... more points? ...

This will help developers to set up sensible environments. I now know
I have been writing autopkgtest scripts that don't assume this and
therefore don't properly replicate a typical desktop environment. But
without these assumptions made explicit, it will be hard to guess what
can be assumed and what can't be.

Best wishes,

Julian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thu, Apr 28, 2022 at 02:32:49PM +0100, Simon McVittie wrote:

On Thu, 28 Apr 2022 at 14:02:46 +0100, Julian Gilbey wrote:

On Thu, Apr 28, 2022 at 10:16:30AM +0100, Simon McVittie wrote:

According to the autopkgtest spec[1],

Tests can expect that the $HOME environment variable to be set to
a directory that exists and is writeable by the user running the test.

That's interesting; I don't find any code that sets HOME in
autopkgtest.

The purpose of autopkgtest is to run the tests in a realistic environment that resembles a real Debian desktop or server, but it doesn't know much about how the testbed system is set up, so it relies on the virtualization backend or the testbed itself to set HOME appropriately. Ideally, it
should be logging into a fully-working Debian system (perhaps under qemu
or lxc, or even on real hardware via ssh) and running tests there.

As I said on #986665, this seems like either a bug in autopkgtest-virt-schroot, or misconfiguration in how you and Laurent
are configuring it (the schroot profile that you're telling it to use).
[...]

I see; this makes more sense now. So can I suggest that
sbuild-setup(7) explains this a bit more and discusses setting up a
meaningful HOME directory? And perhaps has the things necessary for a meaningful XDG_RUNTIME_DIR set up by default as well?

(Incidentally, given that
typically XDG_RUNTIME_DIR is in the ephemeral /run filesystem, writing
to it would not violate policy 4.9.)

Policy §4.9 says that "Required targets must not attempt to write outside
of the unpacked source package tree", then makes a few exceptions for locations like /tmp. /run is not one of those locations, so writing to
/run is not allowed.

Ah, I didn't read it carefully enough; you are right.

I wonder whether autopkgtest
could consider setting up a temporary directory for this purpose like dh_auto_test does, rather than using the heavy machinery of
libpam-systemd?

No, I don't think it should. The purpose of autopkgtest is to run the
[...]

Understood.

Julian

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Fri, 29 Apr 2022 at 10:43:40 +0800, Paul Wise wrote:

Does that mean that autopkgtests that directly or indirectly use the dbus-run-session tool to setup a temporary D-Bus session are buggy?

There's no single answer to that. For unit-test-style testing, it's
often more convenient to start a session bus that's fully under the control
of the test suite, particularly if the tests need special configuration (rejecting certain D-Bus messages, or setting limits, or making mock
D-Bus services activatable, or something like that).

For tests of the overall system (closer to an integration test), the
fact that the service is genuinely activatable on a real system seems
like something that the autopkgtest framework should not prevent you from testing. You don't necessarily need to actually test this - perhaps your priorities are elsewhere, and 100% test coverage is rarely realistic -
but I think it would be a design flaw if autopkgtest prevented you from
testing it, by forcing you to use a mock XDG_RUNTIME_DIR or a mock home directory or anything else that would interfere with your tests' access
to our real system integration layer.

If you run dbus-run-session without the --config-file option, it does
use the real /usr/share/dbus-1/services, so you can test traditional (non-systemd) session service activation like this. You can't test
systemd service activation this way, because that only works for the dbus-daemon that was launched by systemd and told to integrate with it.

dbus has debian/tests/system-bus which does a brief integration test to
check that the system bus works, but it doesn't yet have an equivalent for dbus-user-session - contributions welcome, if you are interested in
expanding coverage.

autopkgtest tries to set up a "real" system login session by
running su, even if it's already running as the target uid, to make
sure that some of the side-effects of login (in particular a PAM
stack) have already happened; but I suspect this doesn't have full
coverage. autopkgtest-virt-ssh is probably the most "realistic" of the backends, by running test commands under a ssh session that is genuinely
a login entry point (meaning processes have a loginuid and so on),
and one of the things on my increasingly intractable to-do list is to
add a way to launch qemu VMs on-demand and use ssh to log in to them,
rather than using a privileged serial console like virt-qemu does.

In some backends it isn't really feasible to run the tests in a "real"
system login session, so integration tests for things that rely on
the system's session infrastructure might have to have the 'skippable' restriction and detect whether they are running in a login session at
runtime. This is also very likely to be something that some autopkgtest backends just cannot provide: -chroot and -schroot don't have their own
init systems (and neither would a Docker backend, review/finishing of
which is also on my to-do list), so they cannot be expected to run their
own system services.

The other thing is testing graphical tools; should that be done in a
mock X11/Wayland session (using xvfb-run), or a real desktop session?

At the moment a mock X11 session using xvfb-run or a mock
Wayland session using weston --backend=headless-backend.so (see debian/tests/run-with-display in src:gtk4) are the only realistic options
for this.

Ideally it'd be possible to test graphical tools under a real desktop
session, but that's not straightforward to set up even under something
like autopkgtest-virt-qemu (the test framework would have to script
a login to a display manager using techniques similar to openQA, or
configure the display manager to autologin and then reboot to make the autologin happen), and I'm not sure whether this would be feasible to
set up under other backends at all. There's a cost/benefit tradeoff here: running tests under a real desktop session is likely to be more difficult, slower, and less robust than xvfb-run, so we should only do it if the
benefit is worth those costs. In particular, we can are not going to
be able to use this as a QA gate unless someone puts a lot of time and
effort into making it completely reliable.

GNOME's installed-tests convention[1], which is a frequent source of autopkgtests for Debian, *does* call for tests to be run in a real,
active GUI desktop session, although in practice most packages'
installed-tests do not actually need this.

[1] https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests

smcv

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Fri, 29 Apr 2022 at 08:34:48 +0100, Julian Gilbey wrote:

So can I suggest that
sbuild-setup(7) explains this a bit more and discusses setting up a meaningful HOME directory?

I'm sure patches are accepted, but the problem with this is that what you
want for sbuild does not match what you want for autopkgtest. For sbuild,
the environment that most closely resembles our real, production buildds involves the /etc/schroot/buildd profile, and a uid whose home directory
is /nonexistent. For autopkgtest, one of the more permissive profiles
like /etc/schroot/desktop is more realistic, and tests are allowed to
assume that they run as a user with a real home directory.

I would personally recommend one of the better-isolated autopkgtest
backends like -lxc or -qemu for running autopkgtest tests. -qemu doesn't
need root, and there are proposed patches adding backends that use
unprivileged user namespaces (-unshare and Podman), which I should
probably be reviewing instead of replying to this email.

And perhaps has the things necessary for a
meaningful XDG_RUNTIME_DIR set up by default as well?

Having a meaningful XDG_RUNTIME_DIR in your local sbuild setup will result
in a risk of uploading packages that cannot be built on our official
buildds, which do not have a meaningful XDG_RUNTIME_DIR (if I remember correctly, the environment variable is set to a path that is valid outside
the chroot, but that path does not exist inside the chroot). This is particularly true for older debhelper compat levels. Again, what you
want for sbuild and what you want for autopkgtest are unfortunately not
the same.

smcv

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Quoting Simon McVittie (2022-04-29 10:58:34)

On Fri, 29 Apr 2022 at 08:34:48 +0100, Julian Gilbey wrote:

So can I suggest that
sbuild-setup(7) explains this a bit more and discusses setting up a meaningful HOME directory?

I'm sure patches are accepted, but the problem with this is that what you want for sbuild does not match what you want for autopkgtest. For sbuild,
the environment that most closely resembles our real, production buildds involves the /etc/schroot/buildd profile, and a uid whose home directory
is /nonexistent. For autopkgtest, one of the more permissive profiles
like /etc/schroot/desktop is more realistic, and tests are allowed to
assume that they run as a user with a real home directory.

I would personally recommend one of the better-isolated autopkgtest
backends like -lxc or -qemu for running autopkgtest tests. -qemu doesn't
need root, and there are proposed patches adding backends that use unprivileged user namespaces (-unshare and Podman), which I should
probably be reviewing instead of replying to this email.

Alas, https://salsa.debian.org/ci-team/autopkgtest/-/merge_requests/138 was merged 14 hours ago. :)

Thanks!

cheers, josch
--==============€54173894110169419=MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Description: signature
Content-Type: application/pgp-signature; name="signature.asc"; charset="us-ascii"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEElFhU6KL81LF4wVq58sulx4+9g+EFAmJrqbcACgkQ8sulx4+9 g+HFDQ//X3HZpNjSdANlGsUt6VUzfnhN2q/kFRZ8k7YZY9kqzaj+TrSFaMI29uoM bmOl9SVGf25fKtgts9EdgVp3wxzXAj090KUY/JKAzRsJMLz+JBinj/T4MJbj7QFR bEQQ/5t7oBBfQqi+VkUSdclHKG+0NFCpTTrNvOemeqVhBy/wAtvYwUEJvZEUb/bD 0h93q1BOEgt0mAbTrhqCGE7MAJeQqBQk8rq/cW4fmIBsN91dHlWxadngu6sJANd4 XaxdZr2SM3EC0RMaEwtj/ZBqPShKk+7+fxE1i7iCfCdFybXeolIE3og5OKAJ03Kx mbqYMqs6KLpsiORivzmAvyGf/stJWr46FbGHnW884SqgyePhy9kZ8EVvvYtqDndK 7kaUOrFvTq485CttnVA4c9BRA2HkWcix0tPDSgfhMaHx3P2PjmkSvgw8pdjJ56+W iT/NKKXPX+E06HeO2SLWsgTrOeAZmKkwl3oDuI0nqP6oDcyITIlhLgDwD8f1LpL1 4us4wY6S00DhZ9U3tGhqMWxP7sIIzOq0YmO0yBRcTpuVM3enXCJLTcCvsbrgzE2j LaAl5O8X4t4IMQlIvUBayc8n0a1tSzsVC5hVG1M8ph4d3DIMzAlvGFaTOgVg/QUX kKUck22tSJ2dHThtyrrxDPGST35y307FK+yO/q6anXGQKGP3M98=
=gXrn
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)