On 6 Nov 2022, at 08:15, Michał Górny <mgorny@gentoo.org> wrote:
Hi, everyone.
Arch testing's relying on automation a lot these days. Not saying
that's bad, if it improves the state of affairs. However, I have some concerns, based on what I've seen lately.
On top of that, it seems that most of it still relies on proprietary
software and we have no clue how *exactly* it works, and it's really,
really hard to get a straight answer.
So, my questions are:
1. Is "runtime testing required" field being respected? Obviously not
every package can be (sufficiently) tested via FEATURES=test, so we've
added that fields. However, if arch testers just ignore it and push
things stable based on pure build testing...
2. How are kernels being tested? Given the speed with which new gentoo- sources stablereqs are handled, I really feel like "arch testing" there
means "checking if sources install", and have little to do with working kernels.
3. How does the automation handle packages that aren't trivially
installable? I recall that in the past stablereqs were stalled for
months without a single comment because automation couldn't figure out
how to proceed, and nobody bothered reporting a problem.
On top of that, it seems that most of it still relies on proprietary
software and we have no clue how*exactly* it works, and it's really,
really hard to get a straight answer.
Hi, everyone.
Arch testing's relying on automation a lot these days. Not saying
that's bad, if it improves the state of affairs. However, I have some concerns, based on what I've seen lately.
On top of that, it seems that most of it still relies on proprietary
software and we have no clue how *exactly* it works, and it's really,
really hard to get a straight answer.
So, my questions are:
1. Is "runtime testing required" field being respected? Obviously not
every package can be (sufficiently) tested via FEATURES=test, so we've
added that fields. However, if arch testers just ignore it and push
things stable based on pure build testing...
2. How are kernels being tested? Given the speed with which new gentoo- sources stablereqs are handled, I really feel like "arch testing" there
means "checking if sources install", and have little to do with working kernels.
3. How does the automation handle packages that aren't trivially
installable? I recall that in the past stablereqs were stalled for
months without a single comment because automation couldn't figure out
how to proceed, and nobody bothered reporting a problem.
--
Best regards,
Michał Górny
On top of that, it seems that most of it still relies on proprietary
software and we have no clue how *exactly* it works, and it's really,
really hard to get a straight answer.
So, my questions are:
1. Is "runtime testing required" field being respected? Obviously not
every package can be (sufficiently) tested via FEATURES=test, so we've
added that fields. However, if arch testers just ignore it and push
things stable based on pure build testing...
2. How are kernels being tested? Given the speed with which new gentoo- sources stablereqs are handled, I really feel like "arch testing" there
means "checking if sources install", and have little to do with working kernels.
3. How does the automation handle packages that aren't trivially
installable? I recall that in the past stablereqs were stalled for
months without a single comment because automation couldn't figure out
how to proceed, and nobody bothered reporting a problem.
As far as I can tell, there's ONE person relying completely on a
proprietary arch testing system.
Ago, could you comment on this? What's blocking you from open sourcing
your software?
On domenica 6 novembre 2022 14:27:40 CET John Helmert III wrote:
As far as I can tell, there's ONE person relying completely on a proprietary arch testing system.
Ago, could you comment on this? What's blocking you from open sourcing
your software?
Hi,
I already answered in the previous post:
"I still use getatoms.py to fetch 'doable' stablereqs (it is on my todo
to switch to nattka). And I have a script the **simply** does emerge over the list of
the packages. There is nothing obscure in it."
I'm working in arch testing since 2009. In the past I relied on scripts done by someone else
and every time there was an issue I got no response.
At a certain point I decided to make my own script in language I know so I can edit it when
is needed.
Since few years we allow self stabilization from maintainer. Do we know how and with
what they test? No because it is not required.
The requirement for test is that the package you are testing works as expected.
https://wiki.gentoo.org/wiki/Project:AMD64_Arch_Testers#Arch_tester.27s_policy[1]
Agostino
--------
[1] https://wiki.gentoo.org/wiki/Project:AMD64_Arch_Testers#Arch_tester.27s_policy
I would be in favour of stepping up the social contract and actually prohibiting this kind of things, we had that before too, the nattka you mgorny wrote is replacement for old bugzilla bot that was ...
closedsource and perished, though nattka now have way more features than
the old thing ever had.
On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
I would be in favour of stepping up the social contract and actually
prohibiting this kind of things, we had that before too, the nattka you
mgorny wrote is replacement for old bugzilla bot that was ...
closedsource and perished, though nattka now have way more features than
the old thing ever had.
As a user, I think it would be really cool if there was a requirement
that all infra and infra-adjacent stuff was free software.
I feel like I've read that Debian already has something like this. While doing some quick searches I didn't find a full-on requirement, but all
their infra bits I did find were powered by free software. The most
relevant ones being buildd [1] and debci [2]. Additionally, the debci
docs has inctructions on reproducing tests yourself [3] which is a nice
extra IMO.
[1] https://buildd.debian.org
[2] https://ci.debian.net
[3] https://ci.debian.net/doc/file.MAINTAINERS.html
- Oskari
On 7.11.2022 8.07, Oskari Pirhonen wrote:
On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
I would be in favour of stepping up the social contract and actually
prohibiting this kind of things, we had that before too, the nattka you >> mgorny wrote is replacement for old bugzilla bot that was ...
closedsource and perished, though nattka now have way more features than >> the old thing ever had.
As a user, I think it would be really cool if there was a requirement
that all infra and infra-adjacent stuff was free software.
I feel like I've read that Debian already has something like this. While doing some quick searches I didn't find a full-on requirement, but all their infra bits I did find were powered by free software. The most relevant ones being buildd [1] and debci [2]. Additionally, the debci
docs has inctructions on reproducing tests yourself [3] which is a nice extra IMO.
[1] https://buildd.debian.org
[2] https://ci.debian.net
[3] https://ci.debian.net/doc/file.MAINTAINERS.html
- Oskari
I _believe_ ago's tinderbox isn't being paid by the GF _anymore_ due to
this reason, but he keeps it running with his own expenses. I don't mind
this as long as the results are desirable and not phony. I still see a
lot of value in most of ago's work.
It is unfortunate we don't get to see the engine behind and copy it,
since I'd be really interested in using his automated bug search /
report tool. Even tattoo (https://github.com/arthurzam/tattoo) lacks
that at the moment.
-- juippis
On 7 Nov 2022, at 06:07, Oskari Pirhonen <xxc3ncoredxx@gmail.com> wrote:
On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
I would be in favour of stepping up the social contract and actually
prohibiting this kind of things, we had that before too, the nattka you
mgorny wrote is replacement for old bugzilla bot that was ...
closedsource and perished, though nattka now have way more features than
the old thing ever had.
As a user, I think it would be really cool if there was a requirement
that all infra and infra-adjacent stuff was free software.
I feel like I've read that Debian already has something like this. While doing some quick searches I didn't find a full-on requirement, but all
their infra bits I did find were powered by free software. The most
relevant ones being buildd [1] and debci [2]. Additionally, the debci
docs has inctructions on reproducing tests yourself [3] which is a nice
extra IMO.
On Mon, Nov 7, 2022 at 6:16 PM Sam James <sam@gentoo.org> wrote:
On 7 Nov 2022, at 06:07, Oskari Pirhonen <xxc3ncoredxx@gmail.com> wrote:
On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
I would be in favour of stepping up the social contract and actually
prohibiting this kind of things, we had that before too, the nattka you >> mgorny wrote is replacement for old bugzilla bot that was ...
closedsource and perished, though nattka now have way more features than >> the old thing ever had.
As a user, I think it would be really cool if there was a requirement that all infra and infra-adjacent stuff was free software.
I feel like I've read that Debian already has something like this. While doing some quick searches I didn't find a full-on requirement, but all their infra bits I did find were powered by free software. The most relevant ones being buildd [1] and debci [2]. Additionally, the debci docs has inctructions on reproducing tests yourself [3] which is a nice extra IMO.
Gentoo has https://www.gentoo.org/get-started/philosophy/social-contract.html.
I feel like something like a dev-run tinderbox is a bit out of the
scope of that.
Suppose I file a bug against a package, pointing out some issue in it.
How do you know I didn't use some proprietary static code analysis
tool to discover that error? Does it even really matter? The bug
speaks for itself. It is like worrying about whether somebody who
filed a bug was running Windows or another proprietary OS or browser
on their desktop.
Well, a tinderbox is just an automated process for doing just that.
We don't require any dev to use a proprietary tinderbox before
committing. It is something that individual devs choose to use for themselves, automating the testing workflow and possibly the
submission of bugs.
I think the key is something that was brought up earlier in the
thread: is this causing problems? If somebody is running some tool
against the repository and automatically filing bugs, and those bugs
are not useful/actionable and waste the time of volunteers, then that
is a problem. Proprietary tools do contribute to this since they can generate results that are harder to reproduce, but if they are clear
and accurate and actionable it could still be a net-positive.
Of course if somebody wants to contribute to 100% FOSS tinderbox
efforts that would be even better. Perhaps if our 100% FOSS tinderbox efforts addressed our needs very well, then nobody would want to
bother with the proprietary reports, or generating them. IMO it would
be better to create the FOSS solution before abandoning the
proprietary one. Doing otherwise is basically burning bridges - it
can be motivating in a sense but not really ideal. I'd love to have a
100% FOSS solution around all of this, but I appreciate what has been
created and can hardly criticize volunteers for failing to make it
happen, especially since I haven't contributed to that myself.
--
Rich
On Mon, Nov 07, 2022 at 07:23:33PM -0500, Rich Freeman wrote:
Proprietary tools do contribute to this since they can
generate results that are harder to reproduce, but if they are clear
and accurate and actionable it could still be a net-positive.
In some cases, yes, this is exactly the problem. This was one of the
bugs reported in the now-deleted issue tracking repository on Github.
On 8 Nov 2022, at 00:23, Rich Freeman <rich0@gentoo.org> wrote:
On Mon, Nov 7, 2022 at 6:16 PM Sam James <sam@gentoo.org> wrote:
On 7 Nov 2022, at 06:07, Oskari Pirhonen <xxc3ncoredxx@gmail.com> wrote: >>>
On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
I would be in favour of stepping up the social contract and actually
prohibiting this kind of things, we had that before too, the nattka you >>>> mgorny wrote is replacement for old bugzilla bot that was ...
closedsource and perished, though nattka now have way more features than >>>> the old thing ever had.
As a user, I think it would be really cool if there was a requirement
that all infra and infra-adjacent stuff was free software.
I feel like I've read that Debian already has something like this. While >>> doing some quick searches I didn't find a full-on requirement, but all
their infra bits I did find were powered by free software. The most
relevant ones being buildd [1] and debci [2]. Additionally, the debci
docs has inctructions on reproducing tests yourself [3] which is a nice
extra IMO.
Gentoo has https://www.gentoo.org/get-started/philosophy/social-contract.html.
I feel like something like a dev-run tinderbox is a bit out of the
scope of that.
Suppose I file a bug against a package, pointing out some issue in it.
How do you know I didn't use some proprietary static code analysis
tool to discover that error? Does it even really matter? The bug
speaks for itself. It is like worrying about whether somebody who
filed a bug was running Windows or another proprietary OS or browser
on their desktop.
On 7 Nov 2022, at 06:07, Oskari Pirhonen <xxc3ncoredxx@gmail.com> wrote:
On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
I would be in favour of stepping up the social contract and actually
prohibiting this kind of things, we had that before too, the nattka you
mgorny wrote is replacement for old bugzilla bot that was ...
closedsource and perished, though nattka now have way more features than >> the old thing ever had.
As a user, I think it would be really cool if there was a requirement
that all infra and infra-adjacent stuff was free software.
I feel like I've read that Debian already has something like this. While doing some quick searches I didn't find a full-on requirement, but all their infra bits I did find were powered by free software. The most relevant ones being buildd [1] and debci [2]. Additionally, the debci
docs has inctructions on reproducing tests yourself [3] which is a nice extra IMO.
Gentoo has https://www.gentoo.org/get-started/philosophy/social-contract.html.
...
That had two parts:
1. https://github.com/projg2/nattka/issues/72 & https://github.com/projg2/nattka/pull/73 (done)
2. https://github.com/arthurzam/tattoo/issues/1 (not done)
Of course if somebody wants to contribute to 100% FOSS tinderbox
efforts that would be even better. Perhaps if our 100% FOSS tinderbox efforts addressed our needs very well, then nobody would want to
bother with the proprietary reports, or generating them. IMO it would
be better to create the FOSS solution before abandoning the
proprietary one. Doing otherwise is basically burning bridges - it
can be motivating in a sense but not really ideal. I'd love to have a
100% FOSS solution around all of this, but I appreciate what has been
created and can hardly criticize volunteers for failing to make it
happen, especially since I haven't contributed to that myself.
I _believe_ ago's tinderbox isn't being paid by the GF _anymore_ due to
this reason, but he keeps it running with his own expenses. I don't mind
this as long as the results are desirable and not phony. I still see a
lot of value in most of ago's work.
It is unfortunate we don't get to see the engine behind and copy it,
since I'd be really interested in using his automated bug search /
report tool. Even tattoo (https://github.com/arthurzam/tattoo) lacks
that at the moment.
If the code was
public, I could try figuring it out and perhaps even fixing it.
On Mon, Nov 7, 2022 at 6:16 PM Sam James <sam@gentoo.org> wrote:
On 7 Nov 2022, at 06:07, Oskari Pirhonen <xxc3ncoredxx@gmail.com> wrote:
On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
I would be in favour of stepping up the social contract and actually prohibiting this kind of things, we had that before too, the nattka you mgorny wrote is replacement for old bugzilla bot that was ... closedsource and perished, though nattka now have way more features than
the old thing ever had.
As a user, I think it would be really cool if there was a requirement that all infra and infra-adjacent stuff was free software.
I feel like I've read that Debian already has something like this. While doing some quick searches I didn't find a full-on requirement, but all their infra bits I did find were powered by free software. The most relevant ones being buildd [1] and debci [2]. Additionally, the debci docs has inctructions on reproducing tests yourself [3] which is a nice extra IMO.
Gentoo has https://www.gentoo.org/get-started/philosophy/social-contract.html.
[...]
I think the key is something that was brought up earlier in the
thread: is this causing problems?
Hi,
Whatever outside the arch testing (like tinderbox) is off topic here since it
is a completely different argument.
To make John Helmert III happy, I just switched to tatt; so my actual workflow is tatt + nattka and there is nothing more.
If there are unanswered questions about the arch testing, please let me know.
Agostino
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 491 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 101:21:11 |
| Calls: | 9,681 |
| Calls today: | 2 |
| Files: | 13,725 |
| Messages: | 6,174,906 |