• Re: Upstreams with "official" tarballs differing from their git

    From Daniel =?utf-8?Q?Gr=C3=B6ber?=@21:1/5 to All on Sat Feb 15 12:40:01 2025
    XPost: linux.debian.maint.ocaml.maint

    On Sat, Feb 15, 2025 at 12:10:06PM +0100, Stéphane Glondu wrote:
    I fixed the elpi package by using something a bit hackish: I added git
    as dep, and if I don't see a .git in the build directory, I create one!

    Here is what the execute_before_dh_auto_build target does:
    if test -f .git; then \
    @echo "Found .git, ok"; \
    else \
    touch .false_git; \
    git init --initial-branch=main; \
    git config user.email "foo@bar.baz"; \
    git config user.name "Foo Bar"; \
    git add dune; \
    git commit -m foo; \
    git tag -a v$(DEB_VERSION_UPSTREAM) -m foo; \
    fi

    and here is what the execute_after_dh_auto_clean does:
    if test -f .false_git; then \
    rm -rf .false_git .git; \
    fi

    This is ugliness to paper over ugliness. Please do not!

    My approach to this specific problem would be to add to dune the possibility to use some configuration file (or environment variables) as input for the substitutions, instead of directly querying the VCS. This configuration
    could then be set up as part of the Debian packaging.

    FYI: If all upstream wants is git metadata I like to introduce them to the wonderful, but obscure, git `export-subst` feature. See git-attributes(1).

    Works with forges, git-archive and everything.

    Example: https://github.com/YosysHQ/yosys/commit/222e7a2da345f01980d9261c40c5d50eced4f9ab
    thoug this was later improved by others https://github.com/YosysHQ/yosys/commit/9d15f1d6ac4a9ff2e1f87cda8c366659027fb76f

    If that's not enough can you point us to what this upstream is doing exactly?

    --Daniel

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCAAdFiEEV6G/FbT2+ZuJ7bKf05SBrh55rPcFAmewe2cACgkQ05SBrh55 rPcYkg//dXrxZUbH/wLMsKzEPw+A9OnsqL6vbZARBH4du4qvl4DEJVlMl7y9Kbq+ 1csxWvEb3cygMC1de5SQykJ1J2fEZFYNFk3tmN8s28acCBdw54USfz9tQ9pu1xI7 9+aK4eu89S13bvoaUi7TSyLjQZub+F2gsdPMGdM8vdroEP9Ghq7F52b/BtdkYvjW 1hzxezsJifGWuwEPfwUHKbvxR5Vx7cTg97F6Wg1UVP65HmVttYmpyZvGZU+2RKba MvgfqzgJjuvjkiWZ8fv5nJqqS7kfV8k5hv+3o1l2hIAD7TpAjozI7FCZaHOAeqPR wCHnJMbehrnMHBxMurgWv23OylcnxRg30xeDeHZ1ZLOG/V4jnXeMGXH9/fY1KNHg P88AtS42loz64QixNbXVATI5m5tEPkFQpLK6Rs2rAPggSvJkF+EebpQEQc8ujwyR EcceEjSfbUb3Bneebfumqbdao297Ggxneqkb6U0f4TF9Dd9jdfPyL9PAm9NGQ9UR A6nQMQdY2F0c9OIIjSvlz1NfEWgpc0Y8Iu3rVWCcB3Cv4CXLxpyR9pmNbFZSrQJI zCGH8imVzAifCTRq8UOmDzs0i4/ckS9jDKsnnclKhB2YB+1Q/+K5oAIo+8won/hk FtN+LbNozqYBFlZmi7Jx7XZeX251YmjP/qcReQ7lFBF54fnfqZ0=
    =Jb5v
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?Q?St=C3=A9phane_Glondu?=@21:1/5 to All on Sat Feb 15 12:20:01 2025
    XPost: linux.debian.maint.ocaml.maint

    Hi,

    Le 15/02/2025 à 10:08, Julien Puydt a écrit :
    Usually, projects using this trick publish tarballs with
    substitutions
    applied:

       https://github.com/LPCIC/elpi/releases/tag/v2.0.7

    (here, elpi-2.0.7.tbz).

    I realize my previous email was a bit short: I was wondering if this
    .tbz still source code because in the autotools world, package sources
    come with configure scripts ready to run, but the good practice in
    Debian is to regenerate those from configure.ac.

    Well, we enter a philosophical debate that is not specific to OCaml and probably should be discussed elsewhere... Adding debian-devel to get
    more opinions.

    Summary to other debian-devel readers: we are facing some upstreams that publish "official" tarballs that differ from what is in their git. The differences may include: variable substitutions, generated files... I
    guess this is pretty common (cf. autotools). Moreover, the build system
    behaves differently if it is called from git or not, or from extracted
    official tarballs or not.

    IMHO, traditionnaly, "source code" from Debian point of view is whatever upstream releases as "official" tarballs (i.e. elpi-2.0.7.tbz), which
    may differ from what is in upstream git (i.e. v2.0.7.tar.gz). What makes
    me think that is the special care that is taken in keeping upstream
    tarballs pristine (with their signatures...).

    Some may consider that this Debian notion of "source code" differs from
    the GNU "preferred form of modification", which would rather be what is
    in upstream git... or is it? In Debian, we "modify" upstreams by
    applying patches on top of them, so I argue that using "official"
    tarballs is fine as long as patches used in Debian packaging apply on
    upstream git as well.

    Anyway, I do think the "GNU" source should be recoverable from the
    "Debian" source. Technically, this is usually not the case with variable substitutions, but IMHO it's acceptable to use the substituted sources
    most of the time.

    On the other hand, insisting on using upstream VCS contents can lead to
    ugly hacks in Debian packaging, such as what you are describing. I must
    admit I usually use "official" tarballs to avoid these hacks (and maybe
    a little out of laziness).

    I fixed the elpi package by using something a bit hackish: I added git
    as dep, and if I don't see a .git in the build directory, I create one!

    Here is what the execute_before_dh_auto_build target does:
    if test -f .git; then \
    @echo "Found .git, ok"; \
    else \
    touch .false_git; \
    git init --initial-branch=main; \
    git config user.email "foo@bar.baz"; \
    git config user.name "Foo Bar"; \
    git add dune; \
    git commit -m foo; \
    git tag -a v$(DEB_VERSION_UPSTREAM) -m foo; \
    fi

    and here is what the execute_after_dh_auto_clean does:
    if test -f .false_git; then \
    rm -rf .false_git .git; \
    fi

    This is ugliness to paper over ugliness. Please do not!

    My approach to this specific problem would be to add to dune the
    possibility to use some configuration file (or environment variables) as
    input for the substitutions, instead of directly querying the VCS. This configuration could then be set up as part of the Debian packaging.

    I suppose dh's ocaml_dune building tool could do that trick itself
    (using debian/fake_dot_git instead of .false_git, better user.email/user.name...), and that would make sure we don't break.

    In case it is not clear: I will oppose this ending up in dh-ocaml.
    However, dh-ocaml would be the right place to add support for the
    approach I outlined above.

    What do you think about the topic?

    My e-mail is very opinionated, I would really like to hear other opinions.


    Cheers,

    --
    Stéphane

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From kpcyrd@21:1/5 to quoting from an email I on Sat Feb 15 13:50:01 2025
    On 2/15/25 12:10 PM, Stéphane Glondu wrote:
    I realize my previous email was a bit short: I was wondering if this
    .tbz still source code because in the autotools world, package sources
    come with configure scripts ready to run, but the good practice in
    Debian is to regenerate those from configure.ac.

    Well, we enter a philosophical debate that is not specific to OCaml and probably should be discussed elsewhere... Adding debian-devel to get
    more opinions.

    Summary to other debian-devel readers: we are facing some upstreams that publish "official" tarballs that differ from what is in their git. The differences may include: variable substitutions, generated files... I
    guess this is pretty common (cf. autotools). Moreover, the build system behaves differently if it is called from git or not, or from extracted official tarballs or not.

    IMHO, traditionnaly, "source code" from Debian point of view is whatever upstream releases as "official" tarballs (i.e. elpi-2.0.7.tbz), which
    may differ from what is in upstream git (i.e. v2.0.7.tar.gz). What makes
    me think that is the special care that is taken in keeping upstream
    tarballs pristine (with their signatures...).

    [...]

    What do you think about the topic?

    My e-mail is very opinionated, I would really like to hear other opinions.

    hello! ✨

    disclaimer upfront, I know pretty much nothing about ocaml, this is
    based on my experience with C/Rust/Go/etc.

    I think the concept of "building the source code into source code" [sic]
    that is common with autotools, is just the regular build in a trenchcoat
    and should happen on Debian build servers. This is to avoid forcing a
    gap between the VCS and Reproducible Builds that nobody feels
    responsible for. Coincidentally this topic was also discussed in #reproducible-builds irc yesterday.

    With regards to signatures, quoting from an email I wrote briefly after
    the XZ incident[0]:

    It's from the old mindset of code signing being the only way of
    securely getting code from upstream. Recent events have shown (instead
    of bothering upstream for signatures) it's much more important to have
    clarity and transparency what's in the code that is compiled into
    binaries and executed on our computers, instead of who we got it from.
    The entire reproducible builds effort is based on the idea of the source
    code in Debian being safe and sound to use.

    [0]: https://lists.debian.org/debian-devel/2024/04/msg00125.html

    I know Debian attempts to regenerate the autotools files, but there is
    no way to tell if this actually worked, I vaguely remembered XZ was specifically one of the cases where it didn't.

    In other news, note there's currently a push within Arch Linux to move
    away from upstream custom tarballs towards VCS snapshots:

    https://gitlab.archlinux.org/archlinux/rfcs/-/merge_requests/46

    Also because people found this interesting yesterday, Arch Linux and
    Debian disagree on "what's the source code of curl 8.12.1":

    Arch Linux: https://whatsrc.org/artifact/sha256:146d2d673358b7927d9a3c74e22b6b0e7f9a1aee2a4307afbe6ac07f12764130
    Debian: https://whatsrc.org/artifact/sha256:599ff98cbab933a8b3640a084b12a5308a20795c192855ee454a8c1c16fa4dac

    Diff between those two:

    https://whatsrc.org/diff-right-trimmed/sha256:146d2d673358b7927d9a3c74e22b6b0e7f9a1aee2a4307afbe6ac07f12764130/sha256:599ff98cbab933a8b3640a084b12a5308a20795c192855ee454a8c1c16fa4dac

    Even if we got some kind human to review the source code in entirety for
    us, which one should they review? sha256:146d2d673358b7927d9a3c74e22b6b0e7f9a1aee2a4307afbe6ac07f12764130? sha256:599ff98cbab933a8b3640a084b12a5308a20795c192855ee454a8c1c16fa4dac?
    Both?

    cheers,
    kpcyrd

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jeremy Stanley@21:1/5 to All on Sat Feb 15 15:30:01 2025
    On 2025-02-15 12:33:16 +0100 (+0100), Daniel Gröber wrote:
    [...]
    FYI: If all upstream wants is git metadata I like to introduce them to the wonderful, but obscure, git `export-subst` feature. See git-attributes(1).

    Works with forges, git-archive and everything.

    Example: https://github.com/YosysHQ/yosys/commit/222e7a2da345f01980d9261c40c5d50eced4f9ab
    thoug this was later improved by others https://github.com/YosysHQ/yosys/commit/9d15f1d6ac4a9ff2e1f87cda8c366659027fb76f

    If that's not enough can you point us to what this upstream is doing exactly?

    I'm not that particular upstream, but with my upstream hat for other
    projects on, there's a lot of data in a Git repository that our
    source tarball build process relies on but isn't strictly files in
    the checked-out worktree: names and addresses of all commit authors,
    most recent tag in the checkout's history and number of commits
    following it, presence of certain footer lines in commit messages
    after the most recent tag, tags in the checkout's history matching a
    specific pattern which come immediately after the introduction of
    each file in a certain directory... these are things easily queried
    from a (non-shallow) clone of the repository but which aren't simple
    string substitutions.

    There are several reasons for this complexity: First and foremost,
    when these projects started Git was still fairly new on the scene,
    and most distros preferred or even required source tarballs for
    packaging. Second, the projects' maintainers were burned on multiple
    occasions by mistakes where metadata duplicated from Git committed
    into the file tree ended up out of sync or straddling release
    points, so developed ways to avoid the duplication and risk of
    divergence by extracting that data from Git at dist build time.
    Third, because the projects needed to deal with heavy volumes of
    development activity from many contributors in parallel, they relied
    on a distributed parallel approval model with the fewest possible
    coordination chokepoints, so needed to support independent features
    merging in arbitrary order with things like release notes sorted out automatically whenever a release got tagged.

    I would argue that our source tarballs don't exactly "differ" from
    what's in Git; they include content which isn't solely represented
    by the worktree files in a corresponding checkout, but is still data
    extracted from the corresponding Git repository state. Downstream
    distros can choose to use our official signed release source
    tarballs, or run the tarball build process themselves from a full
    checkout of our Git repositories, but just naively dumping the file
    tree from a git checkout or even using a shallow clone is inadequate
    and we expressly do not support those workflows (if someone insists
    on doing that, it's on them to make it work, and to check that
    they're not omitting things the copyright license references such as
    a generated authors file).

    Unfortunately, package maintainers sometimes like to insist that
    upstream projects' workflows are "wrong" because the choices they've
    made differ from how other projects might choose to develop
    software, but communities are unique and often face different
    challenges that need their own solutions or aren't willing to
    compromise by adopting partial solutions popular elsewhere just to
    conform.
    --
    Jeremy Stanley

    -----BEGIN PGP SIGNATURE-----

    iQKTBAABCgB9FiEEl65Jb8At7J/DU7LnSPmWEUNJWCkFAmewowlfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDk3 QUU0OTZGQzAyREVDOUZDMzUzQjJFNzQ4Rjk5NjExNDM0OTU4MjkACgkQSPmWEUNJ WCkq6xAAoF2trs//sm30qtJRU43ylCfGrJxk7TDiXP1fk7Kiuz9o7JTfZzfUVflg W/ObyVn0wiapuMScE+ArWWngNvm65qwrUcxMnD9kiah4KiYLZretZ6eI77PAjCy5 0kcA3ScCnSUeIA4i2l+9U54CyBHUt4/NngSZsNLQdKUA9/nz1WHpTo4sH20LOtXC Z25UnxnWzl0rXQiX6AklYjioddOee3TGq0HDzbhRexZ16g+MFAufv4jXAKo8egll U5kvNUwxtmkUPFf0PK377QVMQ0bGCjxCmdf1CZyKbEcZyDCeJ30ecNQT+1QbE6ZL +zivPWetWrur0pOvX/8dJ31lZHwrRXDoj6fLcIY3EWRKlmJi5IapIBRlLGAxw7PH mpvWmXFE7bpCdBJs5dGn5Tsb+QoaZUcHrvKAsYZRNN5vUg+beBt+Hu1/P3n11fWL Xkrbs5qlWzJe3OPsSEB1hnoi1MwFg+DdXI5nI0Y7IB8f0SBwx6irsae2zdTiRECm b4G3nLDkyFXGB2p9086TXcPmNjs9HxZGN5R4bFzqPJTmpPXU7acDjz91mQUfr9le 4STu3QUAD5xCSlp2ndKR1ZPdp4lEMh8ygNwKf7iifjs+DnNjTDstJkiOzPlFiMT0 ju0nCMZp8mJYsgYubEdLVLZKEqzRTgF2yOf4/xbHc1opntOKFWM=
    =5N/B
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32
  • From =?UTF-8?Q?Julien_Plissonneau_Duqu=C@21:1/5 to All on Sat Feb 15 17:00:01 2025
    Hi,

    Le 2025-02-15 13:39, kpcyrd a écrit :
    Diff between those two:

    https://whatsrc.org/diff-right-trimmed/sha256:146d2d673358b7927d9a3c74e22b6b0e7f9a1aee2a4307afbe6ac07f12764130/sha256:599ff98cbab933a8b3640a084b12a5308a20795c192855ee454a8c1c16fa4dac

    That divergence is reflected between the official upstream tarballs and
    the tarballs that can be downloaded from their GitHub releases page.

    There is a similar issue with Gradle where there are official "source distributions" ZIPs [1] (generated by gradle as part of its build) and not-really-that-much-less-official release tarballs [2] automatically
    generated by GitHub. The divergence between both is shown at [3].

    In the case of Gradle, as the official source ZIPs are lacking
    documentation files that I think are important (e.g. LICENSE) I switched
    to using GitHub tarballs, and will probably switch again to use uscan's mode=git as it's simpler to configure and more reliable for large
    projects that release often. That mode could end up being adopted by the
    Java Team for all projects hosted on GitHub for that reason, and also
    because it seems easier to convince upstream projects to sign their
    release tags than to sign their release tarballs. In this specific case
    however I also believe that the upstream project should fix their source archives generation.

    In the case of curl, I believe that for Debian using the official
    tarball is fine. If I had to use their upstream git repo for Debian
    packaging, I would probably write a small custom script that runs their
    maketgz script and can be used in the watch file to regenerate
    official-like "original" tarballs the same way as upstream.

    Le 2025-02-15 12:10, Stéphane Glondu a écrit :

    My approach to this specific problem would be to add to dune the
    possibility to use some configuration file (or environment variables)
    as input for the substitutions, instead of directly querying the VCS.
    This configuration could then be set up as part of the Debian
    packaging.

    I think that this is a reasonable approach that would make the build a
    lot less brittle, and that should be submitted upstream.

    Cheers,


    [1]: https://services.gradle.org/distributions/
    [2]: https://github.com/gradle/gradle/tags
    [3]: https://salsa.debian.org/jpd/gradle/-/blob/287ae5c99790f266e242964321955f7c77f397df/debian/wip/delta-ghtar-gradlezip

    --
    Julien Plissonneau Duquène

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Marco d'Itri@21:1/5 to glondu@debian.org on Sat Feb 15 18:50:02 2025
    XPost: linux.debian.maint.ocaml.maint

    On Feb 15, Stéphane Glondu <glondu@debian.org> wrote:

    On the other hand, insisting on using upstream VCS contents can lead to ugly hacks in Debian packaging, such as what you are describing. I must admit I usually use "official" tarballs to avoid these hacks (and maybe a little out of laziness).
    In my own packages I am happy to add as many ugly hacks are needed to be
    able to directly use the upstream repository as upstream. e.g.: varnish.

    --
    ciao,
    Marco

    -----BEGIN PGP SIGNATURE-----

    iHUEABYIAB0WIQQnKUXNg20437dCfobLPsM64d7XgQUCZ7DTQgAKCRDLPsM64d7X gZogAP9hB+63abb+r+vAuxTQf4u3FUDznLgpLZlj1f4BCb9/SgEA120gnXuHNWVd QghAepraf0geM8Sl4wQVs3L1RIFN5A0=
    =94xk
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Sean Whitton@21:1/5 to All on Sun Feb 16 03:50:01 2025
    XPost: linux.debian.maint.ocaml.maint

    Hello,

    On Sat 15 Feb 2025 at 12:10pm +01, Stéphane Glondu wrote:

    Summary to other debian-devel readers: we are facing some upstreams that publish "official" tarballs that differ from what is in their git. The differences may include: variable substitutions, generated files... I guess this is pretty common (cf. autotools). Moreover, the build system behaves differently if it is called from git or not, or from extracted official tarballs or not.

    IMHO, traditionnaly, "source code" from Debian point of view is whatever upstream releases as "official" tarballs (i.e. elpi-2.0.7.tbz), which may differ from what is in upstream git (i.e. v2.0.7.tar.gz). What makes me think that is the special care that is taken in keeping upstream tarballs pristine (with their signatures...).

    Some may consider that this Debian notion of "source code" differs from the GNU "preferred form of modification", which would rather be what is in upstream git... or is it? In Debian, we "modify" upstreams by applying patches
    on top of them, so I argue that using "official" tarballs is fine as long as patches used in Debian packaging apply on upstream git as well.

    Anyway, I do think the "GNU" source should be recoverable from the "Debian" source. Technically, this is usually not the case with variable substitutions,
    but IMHO it's acceptable to use the substituted sources most of the time.

    On the other hand, insisting on using upstream VCS contents can lead to ugly hacks in Debian packaging, such as what you are describing. I must admit I usually use "official" tarballs to avoid these hacks (and maybe a little out of laziness).

    I think that basing our work on upstream Git makes our source packages
    more useful, and more accurately reflects our commitment to providing
    the preferred form of modification for everything in our archive.

    If our work is based on upstream Git then users can clone source
    packages from salsa (or, better, 'dgit clone' if the maintainer has used
    'dgit push-source') and can use powerful tools like 'git blame' and 'git bisect' to understand their bug.

    With tarballs the granularity of these tools is so much less.

    --
    Sean Whitton

    --=-=-Content-Type: application/pgp-signature; name="signature.asc"

    -----BEGIN PGP SIGNATURE-----

    iQJNBAEBCgA3FiEEm5FwB64DDjbk/CSLaVt65L8GYkAFAmexT/8ZHHNwd2hpdHRv bkBzcHdoaXR0b24ubmFtZQAKCRBpW3rkvwZiQF6lD/9MMYJAS9MMXyv6dT0WgLCS XpohrR/Qb+coVrx449wXicGmwEVZlnUEkWACwCHqBbSIHpkkHPRMwH+iQiuxahyf KYwSi86X2aPhTG63QBjWA8VSk69cKC/Rju8Znp7DGkgk96irfX/bPCUOWj+6bkyA 1MXmyE8sa9KcCwZCuEyeDfVF0nbrMaXwWDzIuNeLOfzzpaEiTVNKmE/M0cOBGAQ1 t/BAX3V49800Ure5HJHgfBops02VgY74mrvSaXzSxAl24zcdJXi2BgUuKprwkmgL iemQ+AeakBdrNcfZEi8J0jTykm8JhynSgcLX8lIUcByLcQAvD8ShUxYD5XEeqEOn pFFljBRuO/jNUuHsqwtsWp+LoXfGC0J6nyklL3uU6x2UomXtXHCk8C4ozXnf9Qmf Pyj0igfCgFR2BGnzPK02REx0fGtPVA6NqRodLvWGG9vI3yrU7exG7ZhHTD2lu3Sw uR02hpsyAnry1oeCcuLungAwFb649EHD3ZD+oKNveBH1m/hgNoAg2Uti9TBrVmyv d44PgmctEVyCDB5MXTbTiH7JaFBqqFMltXQxLFm9jWFtCCfg1Yv/U4nM8zxlq5cy mlGmj/xhCYUtSTwoRs4kdz0jCIfLoNL9vdxyhhJ9Yby9k43cS/GmZCv2qxuSoxBy j0a3dn/Dhpx++wnuw9ww1Q==taiU
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Us
  • From Colin Watson@21:1/5 to Sean Whitton on Sun Feb 16 16:00:02 2025
    On Sun, Feb 16, 2025 at 10:39:59AM +0800, Sean Whitton wrote:
    I think that basing our work on upstream Git makes our source packages
    more useful, and more accurately reflects our commitment to providing
    the preferred form of modification for everything in our archive.

    If our work is based on upstream Git then users can clone source
    packages from salsa (or, better, 'dgit clone' if the maintainer has used 'dgit push-source') and can use powerful tools like 'git blame' and 'git bisect' to understand their bug.

    With tarballs the granularity of these tools is so much less.

    This is a false dichotomy, though. It's perfectly possible to use both
    in conjunction with each other, by importing a tarball on top of an
    upstream git tag so that the differences between them are represented by
    a git commit. There are various tools in Debian to help with this.

    --
    Colin Watson (he/him) [cjwatson@debian.org]

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Simon Josefsson@21:1/5 to Julien Puydt on Sun Feb 16 18:40:01 2025
    Julien Puydt <julien.puydt@gmail.com> writes:

    If you use autotools, you start with configure.ac and .in files. If
    upstream prepared the tree (as they generally do), you also get a libtool/configure, etc. From this, you can run ./configure && make &&
    make install ; that will do substitutions in the .in files. And if you
    want to start anew, you can use autoconf/autoheader/whatever to re-
    create the configure/libtool script and then ./configure && make &&
    make install.

    This is a long-standing misunderstanding in the Debian community. There
    is no guarantee that autoconf/autoheader/whatever re-create all
    generated autotools files. In fact, there are several examples of
    situations where they are not re-generated (e.g., modified aclocal *.m4
    files without bumping serial number) and this is intentional upstream
    behaviour from autotools and there are no signs that this will change.

    To be certain to not get pre-generated files, one approach is to not use tarballs with pre-generated content at all, but to insist on packaging autotools projects based on git content; few upstreams commit generated
    files into git so this ought to be the safest approach. Look at
    'inetutils' in Debian for an example. Another is to manually prepare a
    'rm' list of files to remove in debian/rules, or to use debian/copyright Files-Excluded; to remove all generated autotools files.

    /Simon

    -----BEGIN PGP SIGNATURE-----

    iQNoBAEWCAMQFiEEo8ychwudMQq61M8vUXIrCP5HRaIFAmeyIlAUHHNpbW9uQGpv c2Vmc3Nvbi5vcmfCHCYAmDMEXJLOtBYJKwYBBAHaRw8BAQdACIcrZIvhrxDBkK9f V+QlTmXxo2naObDuGtw58YaxlOu0JVNpbW9uIEpvc2Vmc3NvbiA8c2ltb25Aam9z ZWZzc29uLm9yZz6IlgQTFggAPgIbAwULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgBYh BLHSvRN1vst4TPT4xNc89jjFPAa+BQJl/YgIBQkLehFUAAoJENc89jjFPAa+CboA +wUa06RD5e5VTCxvSWtPS75Wq2qBeYGZnf0jvUMxa2n4AP4xkUeAPPnNuMsTm2fs FCDIGaEM2Yn6Vb2huzzT1Fw/BLgzBFySz4EWCSsGAQQB2kcPAQEHQOxTCIOaeXAx I2hIX4HK9bQTpNVei708oNr1Klm8qCGKiPUEGBYIACYCGwIWIQSx0r0Tdb7LeEz0 +MTXPPY4xTwGvgUCZf2IKwUJC3oQqgCBdiAEGRYIAB0WIQSjzJyHC50xCrrUzy9R cisI/kdFogUCXJLPgQAKCRBRcisI/kdFoqdMAQCgH45aseZgIrwKOvUOA9QfsmeE 8GZHYNuFHmM9FEQS6AD6A4x5aYvoY6lo98pgtw2HPDhmcCXFItjXCrV4A0GmJA4J ENc89jjFPAa+GcYA/26YQY05bLtnXiIjTiAzrGQrRXxTHPA8Av7TDFHvIetWAP9s HSoU8OfTwmTiEnGwLlsV7QJclZg3YNz/Ypcp9TqQBrg4BFySz2oSCisGAQQBl1UB BQEBB0AxlRumDW6nZY7A+VCfek9VpEx6PJmdJyYPt3lNHMd6HAMBCAeIfgQYFggA JgIbDBYhBLHSvRN1vst4TPT4xNc89jjFPAa+BQJl/YgwBQkLehDGAAoJENc89jjF PAa+phoA/jrDqIrl/55vUMBhIQv+TP635d2iCTEnyFmbUcP9+gh6APoDsXalVd2c OGxQtSC+TF8PkZMn1TLkJKAjVxr+xx40AgAKCRBRcisI/kdFojghAQD8SH2pgCuF 8m+vwTRVghUyzpPieIFapJ5BdgG63xEm0gEAttaIFl+DXmtiflTOrvE0GggExArm 9TfQ2b2PIDOY+wM=
    =DcHH
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?Q?Julien_Plissonneau_Duqu=C@21:1/5 to All on Sun Feb 16 20:00:01 2025
    Hi,

    Le 2025-02-16 17:00, Julien Puydt a écrit :

    I have always considered the true way to know if you have the source of
    a package is: imagine you're stuck somewhere with a Debian mirror and
    no external link. Could you take over the development of the package at
    hand?

    Written like that it looks dramatic, but in that project there are only
    5 occurences of %%VERSION_NUM%%. That's only 4 too many, and then it
    would not be much different than many other project releases where a
    script (not always provided), a bot (same) or plain old manual
    intervention (not always fully documented) is used to adjust a hardcoded version number in some file somewhere at release time. But it's true
    that if you're stranded somewhere you may have some free time at hand to implement your own layer of templating ^ ^.

    Cheers,

    --
    Julien Plissonneau Duquène

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Sean Whitton@21:1/5 to rhys on Mon Feb 17 01:30:01 2025
    XPost: linux.debian.maint.ocaml.maint

    Hello,

    On Sun 16 Feb 2025 at 06:18am -06, rhys wrote:

    The potential for additional function is not relevant.

    If the upstream intends to distribute it with a tarball, that's the "golden" package that downstream should base code upon.

    The Debian project officially disagrees with you.

    The preferred form for modification, which is what NEW cares about, is determined by upstream's actual practices, not by their fiat.

    We frequently reject packages from NEW because we have minified files;
    we add the source to debian/missing-sources/.

    --
    Sean Whitton

    --=-=-Content-Type: application/pgp-signature; name="signature.asc"

    -----BEGIN PGP SIGNATURE-----

    iQJNBAEBCgA3FiEEm5FwB64DDjbk/CSLaVt65L8GYkAFAmeygd8ZHHNwd2hpdHRv bkBzcHdoaXR0b24ubmFtZQAKCRBpW3rkvwZiQOKBEACGyuc/QIsyUZNgZf7+OCZU ZfFbE+HPQDmOqXCOQYTuVah4lPCJzeXk44s4dHgkCDvdz62XPT9Mmus89ObEfG13 zNQYJ/2KRZt6rih8Ms8L1gjc5+pAtPy2OpFUoztYaBsAfjnWpqqNmkka3mibX8kw mh5b8oJrFMNa0LgwMETwTnolG3b7t+lL+6Wcs0Lf8ksVe1HCA5kR159SpjYn0gTP q8lJ8WxY3pNin6ZDZChjZQnVYnZuY6Ig+nN4OnLzUil4Ypiwah//c/lgkqbKUoKy UFVZFBYauDpPSl4T1+l06LhjmeSMoW8g5K6jjbYYp5g/kcybzbCT70btBy92xgw0 Zirrx3q80E4eudGcKsubBTNQDixWEIBmLZATob7Xz4qygxHZRPTt1tNtUMPIr6G+ 76eSoakf9xOBsCuO4H4cBbaeojQ9NAViJmBUiNfaxpIt+CIkrB/HJAcM7GQI+MSg Bzlj6vOIgqGSC6erIhTsnbdFEIBHtD5v8V1yY7jLTitONNU36IC6jnS7qrHEPEU7 hXzDpNDt5m3yjN1qIjyy+Vq9/eHXm1RSpIV71349mOEv7ZnzZuLpMlz9q4rS0irn Wr069yD5HvChGed8Q8EIveGOw2oZm7e7LgxVzs2eKi1eAjmurLeVOg2ZYbd6Xuqv o7re+qRrs53KQNwUKDKEoQ==S9yN
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Us
  • From =?UTF-8?Q?Julien_Plissonneau_Duqu=C@21:1/5 to All on Mon Feb 17 10:10:01 2025
    Le 2025-02-16 21:03, Julien Puydt a écrit :

    - the previous version used %%VERSION_NUM%% in only three places, the
    new one uses it more, so it broke my previous hack -- ;

    As a matter of best practices, this should probably be defined in a
    single place and not "hardcoded" multiple times with templating.

    - there are other things than the substitutions done by dune when
    compiling the package, which do not break the build, but will break
    some depending packages later on with strange and misleading errors.

    Do you mean here that using the "official" tbz source tarballs for
    builds outside of a git tree will result in these errors? If so, that's
    a serious upstream build tool issue IMO.

    As mentioned somewhere in the thread I proposed to dune upstream a
    simple mechanism to bypass this git reliance issue, which will make
    packaging much cleaner.

    That's probably the way to go here. I would also suggest modifying
    dune-release so the git release tags end up with the substitutions
    already applied, to make it possible to simply export them and build
    them outside of a git tree.

    Cheers,

    --
    Julien Plissonneau Duquène

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jeremy Stanley@21:1/5 to Sean Whitton on Mon Feb 17 16:30:01 2025
    On 2025-02-17 08:25:03 +0800 (+0800), Sean Whitton wrote:
    On Sun 16 Feb 2025 at 06:18am -06, rhys wrote:

    The potential for additional function is not relevant.

    If the upstream intends to distribute it with a tarball, that's
    the "golden" package that downstream should base code upon.

    The Debian project officially disagrees with you.

    The preferred form for modification, which is what NEW cares about, is determined by upstream's actual practices, not by their fiat.

    We frequently reject packages from NEW because we have minified files;
    we add the source to debian/missing-sources/.

    Unfortunately Debian is also very conflicted on this point.

    Debian has, for legal and logistical reasons, decided that it cannot
    distribute upstream Git repositories as its source packages, and
    instead chooses to try to condense upstream's preferred form for
    modification (back) into source tarballs. In some cases, this
    condensing loses data that upstream considers important, even at
    times things referred to in copyright licenses. The irony here is
    that some of those upstreams do publish source tarballs where that
    data is extracted from their Git repositories so it can be included
    correctly, but package maintainers need to be careful to run the
    same source tarball build process for the basis of the Debian source
    package in those cases and not just pretend that the _files_ tracked
    in Git are the same as upstream's preferred form for modification.
    --
    Jeremy Stanley

    -----BEGIN PGP SIGNATURE-----

    iQKTBAABCgB9FiEEl65Jb8At7J/DU7LnSPmWEUNJWCkFAmezU+BfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDk3 QUU0OTZGQzAyREVDOUZDMzUzQjJFNzQ4Rjk5NjExNDM0OTU4MjkACgkQSPmWEUNJ WCmDUxAAyLX4kUkktcuttubh++mMwtDLBrBT2f69Z6eyObxAdssZ9vlCKDBmcvIM 5XbZc7wm3knMjXpCvhM7kn3Z5aKvx+oJexZ77bFD88+0TFBzcXAq2fSWAGXJ0e6Q 2WAVfStZoRfp/PFwMcri/EeLn4/ZDh+6NCd8dy+023gmoG3a7LOVunNF+c+zKLf0 TubpX3I+8pVcVCYFfgZs5b4RITC6mnO5kL5wxLvGR0Hu2Gkg1r4aWhE+FH4iPTT3 pTisngqvxC14+X/j1ARwoGdaOrfb2Jal4hDVAc3jaFGUMJSwriol+06ipjmJQrrI r4yS9sG4IEt50X+r7ht1o0TvcyWuRfibvuXG912wI7H2Lb0Me29ArntLCsSggQo3 iMivc+um6Qv9SQmPpMdd73e/GMCXodaeLoSvm4XJr22q3jOxNPkkEPQv5QPVz+b6 sftCwqc/4sJ2Wie9HNHYh26j5SDzlq7f8VzthdZY1/s+S4x93u4dEhAijgtwxmY5 k8iOxinTF/f5yCvLsBmVNmgdH/TOLJNXB+nfBWNlDhC1pL3mAezPBVFbzz0UP8/Z XGSAz3Zgp/H1I+6CuaGoCu1P5ELhXQYgojLgxCHxEGmdMmCWyokXBZsoi5E533rX P0tpOIRMd1DnmJv3flWZvQY239nSpU3f3OJkT6Y1r98UR+KDMWI=
    =sJ4m
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32
  • From Sean Whitton@21:1/5 to rhys on Tue Feb 18 09:00:01 2025
    XPost: linux.debian.maint.ocaml.maint

    Hello,

    On Mon 17 Feb 2025 at 08:22am -06, rhys wrote:

    "Actual practices" and "by fiat" are no different here.

    The former refers to what they actuall work with, the latter refers to
    what they claim is the preferred form of modification. These can come
    apart.

    They do as they like with their code.

    Yes, just as we may :)

    --
    Sean Whitton

    --=-=-Content-Type: application/pgp-signature; name="signature.asc"

    -----BEGIN PGP SIGNATURE-----

    iQJNBAEBCgA3FiEEm5FwB64DDjbk/CSLaVt65L8GYkAFAme0O8cZHHNwd2hpdHRv bkBzcHdoaXR0b24ubmFtZQAKCRBpW3rkvwZiQAZ7D/wIBWlSgbCqkwssz3u16/Ro 3CLkXt0M2109QezqzvveTfMdhsXEXNwOueYU9zvn2sZ0SGu0t7Bfz4ZSmY5ygKZh euJMJe9iPorcMkQrYf5ylwya6DoATioByknp+NYyZIq0M8XB8L/+OUSHWsFmGZ4U oqrhap1dadgvDr+7kbsq2s5WwoliKa5vzmRFwJWqJGQusifzKaBulS+R2C11LN7N C5QOcZfC2UXMgnYn47k93z7SWSL/UyzmjP7ZbCi3iLKTwB5EAmbSntSCTKM03jHO Ho2zhPnHiiULocpxMAqVd5xao6YU8gkaEBrJP+k8oomUNnaL8vJ1mYT268jQeWvZ 5GcLO3o2b/SSr2oLL4ceKeW9rMmj/Q3eXryfnOE50gxs1SJvqzA2ApOoUQgV9Wtt zflmkTrD9dgxHwSGUk/SqBZ0/GxHHOhNjxOMCFUp8J0iiSaGDPklIM+QVAnoNEFq 87vLN72PYSLT6/ulc0h0q+j9t1b942qMuY4NH8uWLnWX+fUxoqXSNJrNez/uTgBQ oFzdWcBJCyNpFxS/NoLpI4UQDK1Fts7RGTmwrs3dscwrokKc5pQRIxQTw1TN5Z5L hWiDarNz7c0bzBSAAK7wqZYp+TP3fkXa9miwqixbMwdhe/aEa/J/tXkxOPREhIpN k6kx+bbwO2maIUIiAT6SVw==uIQ8
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Us
  • From Simon Richter@21:1/5 to Jeremy Stanley on Tue Feb 18 10:10:02 2025
    Hi,

    On 2/18/25 00:21, Jeremy Stanley wrote:

    Debian has, for legal and logistical reasons, decided that it cannot distribute upstream Git repositories as its source packages, and
    instead chooses to try to condense upstream's preferred form for
    modification (back) into source tarballs.

    If we "cannot", then it is not something we "choose".

    We have several upstreams whose git repositories contain files that may
    be legal for upstream to distribute, but that do not fulfill our,
    stricter, requirements, or where we do not want to ship the files as
    they are for technical reasons, such as when upstream is vendoring old
    versions of libraries.

    We have a toolchain that can handle file exclusions when importing
    source archives from upstream, and these tools output a compressed
    source tarball.

    In theory, they could output a git bundle with the offending files
    removed, but this would not be useful: simply removing the objects makes
    it impossible to import the bundle, and changing the objects changes the hashes, so neither can fulfill the role of "preferred form for
    modification."

    What I'd like to see at some point is "compressed bundle containing a
    single commit or tag" as the orig archive. Verifying this against
    upstream is a little harder than for upstream released tarballs (where I
    can just check if the file in Debian is bitwise identical to the one
    released and possibly signed by upstream), but that can be fixed with appropriate tooling.

    This would not work, however, for upstreams where we exclude files: for
    this, we'd need to extend git to allow us to both create and import
    incomplete bundles. That is a lot harder, but would be required for this
    to be universally useful.

    (there is also some correlation between "upstream's git does not contain
    any unredistributable files or vendored dependencies" and "upstream
    ships a usable source tarball", so precisely the cases where git is the
    only upstream we have are the cases where that is least useful to us)

    Simon

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?Q?Julien_Plissonneau_Duqu=C@21:1/5 to All on Tue Feb 18 10:40:01 2025
    Hi,

    Le 2025-02-17 22:47, Julien Puydt a écrit :

    Their .tbz is working nicely -- but it isn't source anymore!

    I think that we disagree on this one. Here the extent of source code and
    build files changes between the .tbz and the git repository is
    reasonably small and can be compared to what happens in many projects
    where at release time a local release branch is created, variables and
    metadata are adjusted, the sources/binaries/docs distributions are
    built, and only the tag is pushed to the public repository. The issues I
    see in this project is that these changes are not committed to the git repository at all, and that using external templating to set a version
    number in so many different places is an antipattern. If you wanted to
    manually revert these changes to get the source tree in the same state
    as the upstream development branch, it would only take you a few
    minutes. If this really bothers you, you could also ship a diff in your
    debian source package, automate the update of said diff with a custom
    script run by uscan and document all that in d/README.source.

    Here I would probably use the .tbz as I would consider it the preferred
    form of modification when it comes to fixing things in this version of
    the package (less steps and thus less risks that a later release of some
    build tool behaves differently and breaks things).

    Cheers,

    --
    Julien Plissonneau Duquène

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Holger Levsen@21:1/5 to Julien Puydt on Tue Feb 18 15:00:01 2025
    On Mon, Feb 17, 2025 at 10:47:04PM +0100, Julien Puydt wrote:
    Yes. Most upstream have little clue on what a best practice is, and
    need to be explained at length.

    TBH I basically stopped reading here, this assumption is so flawed, the (upstream) world is way more diverse.


    --
    cheers,
    Holger

    ⢀⣴⠾⠻⢶⣦⠀
    ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org
    ⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
    ⠈⠳⣄

    I used to be scared for our grandchildren's future. Such optimism!

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEEuL9UE3sJ01zwJv6dCRq4VgaaqhwFAme0kNwACgkQCRq4Vgaa qhzx4w//RqaV71bMedWm2Ez9OkP6241KquRu/pxf9VVfjsUmxRE2LWcde/nq5z6e rT02FNUPo9HZ3lHJBQWh+EsqyFPSDq2FO3vnOPY4056e+w+Rkwj82esqrWH6djN1 xJPuK4SnBn1uAvsXOKrWahXqqv8XKTyvwHDD3owKdP1KCHIeWkb+ESi532yTyf8V JH1X8V9i/uxTZYtkYEwOXLXRpd3ofsXZq7H7N3r88/552L+zLFar+FnEbfPcAyVX CPLZ2zXS/3WmcyLPN9KHA+FuF7Uc5G7W721JxGYtsYYppVESw7Gh6chola7SAGPI vqFQSnKZcCs9qnjtYDnsG4/vVjOLdoecwX6PqVNLjGjFOGgo8zYn3aultT6iaTrg AT0olGOeK2UYc8qw7Yr2VlW3oCmYoNK3cXAnaNDIYARENSJXqGNrlDFw4B/2WGN4 cwhNuXVXs5l1Eiv7PikW/gi6Jm/0YSWZpKXm4r4i6NmOqJYJE2tKOanx9Ap2dOzN JvxjIrw89iZ/zHaNYgqfkfJZfXQwDHVnF4aqhjsfS9kE4xB6MsARuLtWIg9ow7Kr kvoiG5w4igguog0qg7nQsEB/rVML7oWsMVLcsAkk78OJmy