Hi!

Considering what other Linux distros are doing and what is popular
among software developers today, Debian will probably gain more
contributors and be more approachable if we don't reinvent too many
custom things, and hence...

..

In the current DEP14/DEP18 discussions a lot of discussion was had
about how we should represent Debian things in git; your mail also
goes into this direction.

My *feeling* is we should do the opposite - that is, represent less
Debian stuff in git, and especially do it in less Debian-specific
ways. IOW, no git extensions, no setup with multiple branches that
contain more or less unrelated things, etc.

I think we should move more towards a setup that is easily
understood by people not closely following our Debian-specific
things. We should avoid surprising things, again that would include
the multiple branches and any git extensions.

Before pushing for new ways of representing Debian stuff in git, I
think it would be a good idea to learn from all the other distros
and distro-like systems successfully using git [1]. Debian is not
the only distro that wants to use git to capture changes and
encourage contributions to its packages.

Chris

[1] alpine, homebrew, freebsd ports come to mind immediately. nixos
and others too.

…this is the right attitude and I wanted to cater to it and summarize
how packaging sources look in various distros.

I am using MariaDB in the examples as it is a big and common package
that can be found everywhere.

## Alpine
Example: https://git.alpinelinux.org/aports/tree/main/mariadb
Package defined in a single APKBUILD file somewhat similar to a .spec
file. Patches are plain patches in the same directory. References to
upstream source packages by SHA256SUMs. In my view it is not
effortless nor transparent to track code changes. Hosting is a
monorepo on cgit. The official way to contribute is on their GitLab
instance using Merge Requests at
https://gitlab.alpinelinux.org/alpine/aports

## Arch
Example: https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=android-x86-64-mariadb
Very similar to Alpine, package defined in a single PKGBUILD file.
Patches as plain files in the same directory. No upstream git or
files, just MD5SUM references to upstream tarballs. One git repository
per package. Authoritative git repo on cgit. Collaboration features
via https://gitlab.archlinux.org/archlinux.

## Fedora
Example: https://src.fedoraproject.org/rpms/mariadb10.11/tree/rawhide
Fully custom code hosting, building and testing system due to long
history building up their own infra, which includes a facility to make
pull requests. Base of packaging is a .spec file plus a bunch of
included extra files. Patches as plain files.

## CentOS Stream
Example: https://gitlab.com/redhat/centos-stream/rpms/mariadb/-/tree/c9s Basically, downstream from Fedora with some modifications. Hosting and collaboration on custom GitLab instance.

## Rocky Linux
Example: https://git.rockylinux.org/staging/rpms/mariadb/-/tree/r9
This is one of the CentOS clones that came about after Red Hat
acquired CentOS. Inherits the .spec file from Fedora/CentOS. Custom
GitLab instance for all collaboration.

## OpenSUSE
Example: https://build.opensuse.org/package/show/openSUSE:Factory/mariadb Packaging based on .spec file and individual patch files. The .spec
file format is custom and not fully the same as in the Fedora
ecosystem. Hosting and building on their custom-build platform (the
Open Build System) which includes graphical merge requests under name "Requests".

## NixOS
Example: https://github.com/NixOS/nixpkgs/tree/nixos-unstable/pkgs/servers/sql/mariadb
Custom single-file packaging format in a .nix file. No upstream source
code hosting, just reference by SHA256 to upstream tarballs. Monorepo
with all packaging files. Uses GitHub. Currently, 5k+ open Pull
Requests and almost 300k (!) closed ones.

## Gentoo
Custom single-file .ebuild files, with some patches as individual
files under files/. Authoritative hosting on cgit, but https://packages.gentoo.org/packages/dev-db/mariadb links to GitHub
for Pull Requests as the recommended way to contribute.

## Deepin
Example: https://github.com/deepin-community/mariadb/tree/master/debian/deepin The most popular Debian variant in China. Packaging done in
subdirectory `debian/deepin/`. Hosted on GitHub. Imports seem to be
one commit per upstream release, not sure if the content is from git
or tarball, but it isn't upstream git commits. They seem to have some automation that files Pull Requests automatically when new upstream
import is pending.

## OpenMandriva
Example: https://github.com/OpenMandrivaAssociation/mariadb/tree/rolling
Again a .spec file user, patches as individual files. Hosting and Pull
Requests on GitHub.

## FreeBSD ports
Example: https://www.freshports.org/databases/mariadb114-server/ -> https://cgit.freebsd.org/ports/tree/databases/mariadb114-server
Custom ports directory, package source builds based on Makefiles.
Authoritative hosting on cgit, but they have also Codeberg, GitHub and
GitLab mirrors. Most contributions seem to be patch files attached to
bug reports (e.g.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=274494) but there
are also some GitHub PRs
(https://github.com/freebsd/freebsd-ports/pulls). The GitLab mirror
had zero Merge Requests.

## OpenBSD ports
Example: https://openports.pl/path/databases/mariadb
Custom ports directory. Source code hosted on both csvweb and GitHub
mirror. Monorepo with only packaging files (Makefile + patch files and
more). Instead of upstream sources there are files named 'distinfo'
that contain the upstream tar.gz name and SHA256.

## MacPorts
Example: https://ports.macports.org/port/mariadb-10.11/details/ -> https://github.com/macports/macports-ports/blob/master/databases/mariadb-10.11/Portfile
Custom ports directory software. Packages in custom Portfile in
monorepo with SHA256 references to upstream tarballs.

## Homebrew
Example: https://formulae.brew.sh/formula/mariadb -> https://github.com/Homebrew/homebrew-core/blob/master/Formula/m/mariadb.rb Again, custom ports directory, custom packaging format, files on
GitHub in a monorepo and upstream tarballs referenced by sha256.

## Chocolatey
Example: https://community.chocolatey.org/packages/mariadb -> https://github.com/mkevenaar/chocolatey-packages/tree/master/automatic/mariadb This is the Windows equivalent of Homebrew. Packaging in .nuspec
files, fully custom thing. A centralized directory and seems a lot of
the packages are in a single monorepo on GitHub, not sure if that is
all, though.


Conclusions & personal comments:

- Everyone uses git. Sentiments that the Debian repository is enough
of version control, or that git would somehow not be suitable for
Debian packaging, seem detached from practical reality.

- Everyone uses normal git. The fact that Debian has two or three
branches (debian/latest + upstream/latest + pristine-tar) is a bit
special. I would still consider it warranted, as we have higher
standards regarding hosting the source code and having a full audit
trail of changes. If this is ever simplified, the somewhat obvious
solution would be to introduce a new debian/latest branch structure
that is just the current 'debian/*' contents plus a file, perhaps
called 'upstream-source', with url and sha256 sum of upstream tarball.

- GitHub and GitLab are quite common, people know how to use them. The
fact that we have salsa.debian.org is an asset for us and helps
attract new people. Thanks to various cli-tools people preferring to
do all work from the command-line (or from Emacs) should not
inherently be worse off if we have more contributors joining the
project who use Salsa.

- Pull Requests and Merge Requests are also very common. I think the
best course forward would be to have an open mind in accepting
contributions both as git pull requests as well as patch files in
email.

- The number of contributors/maintainers is low everywhere. Ending single-person maintainership is not going to happen any soon, but
hopefully, we can work towards first increasing the pool of
contributors who participate, and then expand on practices around
Merge Requests and reviews and maybe have some kind of formal
sign-offs from at least two people before upload. Initially, perhaps
only for the top-150 packages. But before we can institute review
workflows, we need to have more unification around the version control
and basic packaging workflows.

- Otto

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi,

Quoting Otto Kekäläinen (2024-08-27 08:42:53)

Before pushing for new ways of representing Debian stuff in git, I think it would be a good idea to learn from all the other distros and distro-like systems successfully using git [1]. Debian is not the only distro that wants to use git to capture changes and encourage contributions to its packages.

Chris

[1] alpine, homebrew, freebsd ports come to mind immediately. nixos
and others too.

…this is the right attitude and I wanted to cater to it and summarize
how packaging sources look in various distros.

thank you for your investigations.

- The number of contributors/maintainers is low everywhere. Ending single-person maintainership is not going to happen any soon, but hopefully, we can work towards first increasing the pool of contributors who participate, and then expand on practices around Merge Requests and reviews and maybe have some kind of formal sign-offs from at least two people before upload. Initially, perhaps only for the top-150 packages. But before we can institute review workflows, we need to have more unification around the version control and basic packaging workflows.

I'm still dubious any "2 people sign-off" can work [1]. In your investigations, did you find other distributions which implemented this successfully?

I think "work towards easier collaboration" and "require more than one person for every commit/upload" are two very different things which should be discussed independently.

Thanks!

cheers, josch

[1] My own experience with this comes from my contributions to devscripts which is in the debian group, thus "team" maintained and probably all of you have it installed and should feel responsible for it (right?). Nevertheless, my MRs mostly get zero replies, so I usually just merge them after waiting a couple of months. The situation is a bit better for sbuild but not by much. --==============„67926856848456353=MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Description: signature
Content-Type: application/pgp-signature; name="signature.asc"; charset="us-ascii"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEElFhU6KL81LF4wVq58sulx4+9g+EFAmbNkroACgkQ8sulx4+9 g+GGGxAAja65OmrPWRyZRo2ja+KLR3FSk8NagaXQgK9nTiCRFZqZ5tciNKVWC6I2 PNnr06hYUmHDke0BdpHW7Ja3xLSJ5vuxalJNjK0lvyQJ0RLYk6eVsK2XK8Oildji b/hkzf1XxRIdSGHqf8s2+f6dmoRoS3doEgrTM8CV//9n+Oe9aXCMPnmmLw6mmAtb 9NP3Bo+qqWA8zIlEOuQH9D24bANt2vsNYgWhYgtIQ29a4ufVff8wXONCDwomT4DK YyPjuOPRKDOdoQsnN+U6pR9AqIWnft0I+NvjZ0+bOyS39TW4ncwrXV00QkkHIrC1 MPVBEd6a0hL54471ACnv8PYPU0gjEmDHJi5A1W3OhUR0XyAWGwR7dCmDmfm+ETb5 V5Ws9C/FHBQK+KqqD2JDCYxzDpQ7Cr8R+FzTdKvwA9UzLh9BdVF2pAR5PpxbeEfu UyVWmPWC6zQOkTe1GBz2z/iwqfWiifn7266SE4LlBMcmWYIPxWfWIhrjUeYNPdxh j7L022FyHIONVMv0GmkpA6qyvNCk/D+zXyO5LVi4q2jCZW4ZwkZAoVilqUlJYwIL fqDepelK0JX3OwlY5QtGLQeXRTVsa1gMvqOzxkiru7d1pwPa9SiLgWdfHECQx//m 4FjFxh1wV/g4PHuHAtASWj6v5cx3FC7qhQuGbCDw7fXyRXqWD1A=
=4Nu3
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi!

...

- The number of contributors/maintainers is low everywhere. Ending single-person maintainership is not going to happen any soon, but hopefully,
we can work towards first increasing the pool of contributors who participate, and then expand on practices around Merge Requests and reviews and maybe have some kind of formal sign-offs from at least two people before
upload. Initially, perhaps only for the top-150 packages. But before we can institute review workflows, we need to have more unification around the version control and basic packaging workflows.

I'm still dubious any "2 people sign-off" can work [1]. In your investigations,
did you find other distributions which implemented this successfully?

I think "work towards easier collaboration" and "require more than one person for every commit/upload" are two very different things which should be discussed independently.

Thanks!

cheers, josch

[1] My own experience with this comes from my contributions to devscripts which
is in the debian group, thus "team" maintained and probably all of you have it
installed and should feel responsible for it (right?). Nevertheless, my MRs mostly get zero replies, so I usually just merge them after waiting a couple of
months. The situation is a bit better for sbuild but not by much.

I see you have 41 MRs at https://salsa.debian.org/debian/devscripts/-/merge_requests?scope=all&state=all&author_username=josch
Some of them were approved by Niels, but indeed most go without
approval nor even comments.

I Salsa CI and in Lintian for example I do see pretty good review
activity. Perhaps we just need to have more visibility and reminders
for people to check out https://salsa.debian.org/debian/devscripts/-/merge_requests or even https://salsa.debian.org/groups/debian/-/merge_requests which has now
993 MRs open (and nearly 10k MRs in total).

I wonder if it would make sense to organize some kind of "office
hours" for code reviews and code testing best practices and
guidance...

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 11/10/24 3:42 AM, Otto Kekäläinen wrote:
[ Requiring two party sign-off on every change ]

I wonder if it would make sense to organize some kind of "office
hours" for code reviews and code testing best practices and
guidance...

As someone working in a very large organization where this is required:
That's fine in a company setting. It is not in a volunteer setting,
unless you introduce very firm expectations about turnaround times. As
much as I would see the benefit of that, Debian with its highly
asynchronous work ethics is not that environment.

It is not a problem of best practices and guidance. It is one of
available time, motivation (spoons), and commitment. In a work place you
can demand that and if necessary, reassign to some other team member
whose job it is to review.

Insider risk concerns aside, maybe we should focus on making mistakes
easier to roll back. That is very heavyweight today. And that many
Debian services do not have appropriate staging environments is also a
problem - orthogonal to code review but related to testing practices.

Kind regards
Philipp Kern

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

[ Requiring two party sign-off on every change ]

I wonder if it would make sense to organize some kind of "office
hours" for code reviews and code testing best practices and
guidance...

As someone working in a very large organization where this is required: That's fine in a company setting. It is not in a volunteer setting,
unless you introduce very firm expectations about turnaround times. As
much as I would see the benefit of that, Debian with its highly
asynchronous work ethics is not that environment.

To me the whole free and open source movement is about collaboration, transparency and the ethos of "given enough eyeballs, all bugs are
shallow". I don't think code reviews is a luxury reserved for
corporate employees, but something we can and should do in Debian.
Given the correct tooling, it should work well (and is already working
well in many teams).

I don't subscribe to turnaround time requirements. We don't have
deadlines like in corporations. We publish code and review when we
have time. Code reviews are asynchornous by nature and not in conflict
with Debian or general open source practices. If somebody wants to
team up and work intensly on something, they can if they want. People
can have miniconfs etc. Sure it requires a bit of patience sometimes -
one of my patches to Redis took 6 years before it was merged, and in
Debian for example I have been waiting for example for
lintian.debian.org to come back online for almost 2 years now - but
that is how volunteers collaborate. Projects that have active
collaborators and good collaboration practices will go further.
Hopefully Debian can have a future with increased collaboration and
teamwork.

It is not a problem of best practices and guidance. It is one of
available time, motivation (spoons), and commitment. In a work place you
can demand that and if necessary, reassign to some other team member
whose job it is to review.

The whole point of DEP18 is to streamline/unify the basic packaging
stuff that is essentially undifferentiated and should not be as
complex and multi-faceted as it is now. That frees up to do more fun
things. To me the fun in open source boils down to design discussions,
running code and code reviews. If you don't have time for code reviews
in Debian, maybe indeed best practices and guidance is needed so you
can do it without feeling it is a burden?

Insider risk concerns aside, maybe we should focus on making mistakes
easier to roll back. That is very heavyweight today. And that many
Debian services do not have appropriate staging environments is also a problem - orthogonal to code review but related to testing practices.

That is an interesting idea. I wonder how it would look like in practice?

The cleanup after the xz issue was simply to revert in git and
reupload. How can such "rollback" work be made easier or faster?
Detection of oddities in upstream tarballs can for sure be made easier
when git-buildpackage is used correctly, and having more
team-maintenance as actual teams with reviews will help prevent
problems slipping through. I can't come up with ideas how to make roll
backs easier without increasing the risk of rollbacks of rollbacks,
but perhaps somebody else can brainstorm this area.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sun Nov 24, 2024 at 4:45 AM CET, Otto Kekäläinen wrote:

[ Requiring two party sign-off on every change ]

I wonder if it would make sense to organize some kind of "office
hours" for code reviews and code testing best practices and
guidance...

As someone working in a very large organization where this is required: That's fine in a company setting. It is not in a volunteer setting,
unless you introduce very firm expectations about turnaround times. As
much as I would see the benefit of that, Debian with its highly asynchronous work ethics is not that environment.

To me the whole free and open source movement is about collaboration, transparency and the ethos of "given enough eyeballs, all bugs are
shallow". I don't think code reviews is a luxury reserved for
corporate employees, but something we can and should do in Debian.
Given the correct tooling, it should work well (and is already working
well in many teams).

The "given enough eyeballs" always happened after the fact though.
People scratched their own itch and wanted to publish the fruits of
their work on their own schedule. See also the single maintainer in
Nebraska meme. Allocating resources to code review means higher
quality results, but probably less results overall.

I don't subscribe to turnaround time requirements. We don't have
deadlines like in corporations.

People are already annoyed at not getting things done. Of course some of
it is also due to us requiring review from a selected few (think NEW, or
DSA for infrastructure changes), rather than the developer base at
large.

We publish code and review when we have time. Code reviews are
asynchornous by nature and not in conflict with Debian or general
open source practices.

Good things come to those who wait. At the same time being able to
quickly make changes when you have time and not context switch away
for weeks until someone else had time to get back to you is a valuable motivator.

I agree that in a team - and not a one-person show - the expectations
can (and probably should) be different.

If somebody wants to team up and work intensly
on something, they can if they want. People
can have miniconfs etc. Sure it requires a bit of patience sometimes -
one of my patches to Redis took 6 years before it was merged, and in
Debian for example I have been waiting for example for
lintian.debian.org to come back online for almost 2 years now - but
that is how volunteers collaborate. Projects that have active
collaborators and good collaboration practices will go further.
Hopefully Debian can have a future with increased collaboration and
teamwork.

Question is if it makes things worse.

The whole point of DEP18 is to streamline/unify the basic packaging
stuff that is essentially undifferentiated and should not be as
complex and multi-faceted as it is now. That frees up to do more fun
things. To me the fun in open source boils down to design discussions, running code and code reviews. If you don't have time for code reviews
in Debian, maybe indeed best practices and guidance is needed so you
can do it without feeling it is a burden?

Or, you know, have other people around with enough spoons.

[ Easier rollbacks ]

The cleanup after the xz issue was simply to revert in git and
reupload. How can such "rollback" work be made easier or faster?
Detection of oddities in upstream tarballs can for sure be made easier
when git-buildpackage is used correctly, and having more
team-maintenance as actual teams with reviews will help prevent
problems slipping through. I can't come up with ideas how to make roll
backs easier without increasing the risk of rollbacks of rollbacks,
but perhaps somebody else can brainstorm this area.

That isn't entirely true. For a brief period we thought we needed to
rollback and then recompile the archive to a state of many months prior.
We would not have known how to do that, especially when it comes to
versioning. The best we have to rollback packages is to push an entirely
new build with a +really version (or an epoch). At no point it is "just
a revert in git" or - the gold standard - a revert to the package built previously (it might also have been miscompiled).

In enterprise environments the answer is pinning >= 1000. And keeping
packages simple enough that downgrades keep working.

I remember a couple of times where the Release team stepped in and
reverted maintainer actions that were unfortunately timed - which comes
with a large risk of conflict.

We have fared ok with the current approach, but it is not an environment
of "rollback first, ask questions later" - the overhead is high.

Kind regards
Philipp Kern

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

"Philipp Kern" <pkern@debian.org> writes:

On Sun Nov 24, 2024 at 4:45 AM CET, Otto Kekäläinen wrote:

To me the whole free and open source movement is about collaboration,
transparency and the ethos of "given enough eyeballs, all bugs are
shallow".

The "given enough eyeballs" always happened after the fact though.
People scratched their own itch and wanted to publish the fruits of
their work on their own schedule. See also the single maintainer in
Nebraska meme.

You can see this "hero mentality" throughout Debian's setup, culture,
and processes - I am curious as to whether there is appetite to change
it?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)