• automated CI on salsa to prepare MR for new upstream version

    From Yaroslav Halchenko@21:1/5 to All on Fri Jan 20 17:10:03 2023
    Dear Debian gurus,

    Lazy me would love to get automation going akin to the one in
    conda-forge (see e.g. [1]) -- whenever a new upstream is released
    (as could be detected with a run of uscan) the CI would

    - would run uscan

    - get new source {tarball}
    - get new {uversion} (from uscan output/tarball)

    - would run

    gbp import-orig -- --pristine-tar --debian-branch=new-{uversion}/$(gbp config buildpackage.debian-branch) ...

    i.e. import sources and prepare all 3 branches with dedicated prefix
    new-{uversion}/

    - might do some other automated/decided upon actions such as e.g.
    "drop all debian/patches/00*" which were cherry-picked from upstream,
    and were applied etc.

    - push those branches into salsa repo and submit a merge request
    against the debian-branch (while providing URLs for other two
    branches).

    - employ salsa-ci-team/pipeline (or alike) to build package from
    those 3 branches

    - maintainer then reviews/tunes up the MR and if merges, CI would
    merge/push all the new-{uversion}/ branches into corresponding
    one and delete them from CI.

    If MR is closed, CI would just remove all those new-{uversion}
    branches.


    Is there any package/tool which already does something like that on CI?
    Or may be you see some issues with such a procedure?

    Thanks in advance!

    [1] https://github.com/conda-forge/datalad-feedstock/pull/95

    --
    Yaroslav O. Halchenko
    Center for Open Neuroscience http://centerforopenneuroscience.org
    Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
    WWW: http://www.linkedin.com/in/yarik


    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCAAdFiEExbkF8OjZ/ZZo/zZvot4jUGLaM/oFAmPKutQACgkQot4jUGLa M/rwsw//THL8dXc19SZjixMxCSahuJWCN6bmqMcrb0UPN97HhVF4AmBWD2Wp3C35 5JIu16DzySAt/sa+dqcW6tGfJiGOK8VNsI2zdriDQk3ZUdmtHJvxoTqa4MxdI7jU P91rSRw+gvWDmjXcY4lQGJHOuZjuDWcGUtm3THZjivCyTkN11aVbKjvDK9UGzkAl GzbvjhWiRfxRS+3iV0YlOQzW7wkU0N854RCvNnNZj+370P2fRnML/9ePaZ8yKPQ4 ktL5jF+VPBlXkjhPMF+zKAVLmHCPFGSKsn0KkwTDdvVgsXRKvO7kYsniDPlQr4oa TvE/VEYRXxuYBLKQ7moUzPH7Ra9ZU7qtRf9MUTGthMyU30YhXX0SS5miNUpzs4wp 8TBHU1visIOulkn62Krc/BGWiNiNzDhcWHokKYyxyQS8syCmz/goqzf70FTCw1V+ rCRfFN5Z/PrijmjLPX8IlPHrkFAAqK6jQVTYwQLkV0a7A/PEb+cXYSCLPM8yRTuS dNrXTmpTSO1KHk9c1MbiblJDnsIHx/U7kAW1N1M8D7rjeMVbqGRriwgt1mRqZU7D jRiDbEA9DC15Ck+9LmR2xT1aKez3wr5eA7H6o8GHhu1uLsDyWhITyBP4xvSCGCTN eSdQkExJaXiEvmj3HY3AgikoJ9iWWbP8paIJjHxnr0
  • From undef@21:1/5 to All on Fri Jan 20 23:10:02 2023
    Lazy me would love to get automation going akin to the one in
    conda-forge (see e.g. [1]) -- whenever a new upstream is released
    (as could be detected with a run of uscan) the CI would


    It's not exactly what you describe (because we're rebasing), but the
    Mobian team has been doing something like this for a while to update our
    kernel packages. The code is at https://salsa.debian.org/Mobian-team/packaging-tools/-/blob/master/rebase-ci.yml

    You'd need to replace the git work in the first few lines with uscan,
    but otherwise:

    - gbp pq should deal with upstreamed patches (and list them)

    - The package will be pushed to a new branch and an MR opened. This
    requires a bot account or access token to be created.

    - After the push/MR, normal salsa jobs will be run. (This one only runs
    on a schedule with a defined variable, see https://salsa.debian.org/Mobian-team/packaging-tools/-/blob/master/rebase-ci.yml#L61)

    - If the patches fail to apply an issue is opened with a link to the
    failed CI job.


    Only real issue we've had with it so far is getting a heap of "failed to
    rebase on 6.1.x" issues raised, but the kernel has a near weekly release
    cycle, so it's probably not as bad elsewhere.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Paul Wise@21:1/5 to Yaroslav Halchenko on Sat Jan 21 00:50:01 2023
    On Fri, 2023-01-20 at 11:01 -0500, Yaroslav Halchenko wrote:

    Is there any package/tool which already does something like that on CI?
    Or may be you see some issues with such a procedure?

    The Debian Janitor is already doing some of that for all packages,
    for both new upstreams and new upstream commits.

    https://janitor.debian.net/fresh-releases/ https://janitor.debian.net/fresh-snapshots/

    --
    bye,
    pabs

    https://wiki.debian.org/PaulWise

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEEYQsotVz8/kXqG1Y7MRa6Xp/6aaMFAmPLJoQACgkQMRa6Xp/6 aaPv1A/8DGVqFc1xOLnOgfpLt1AbjJkLV4aJ3Jmdozpx0AEZ1LMLjykyzTl5UHcI L0qwYNEo6xmagXQyF5KYo1N3DNyQPjkNMZmYdTnxDq6SaPauTiLbHpSxxt3x6L63 23ZPcVlOyXoPv0l+SZvvLQ+w25pvtzk8iY5zuOh+z5KspIa8Sh1pYK4KqeD4c/qF U+h5mZ9rmslyYlXhgmNixvYViHzCeptl3Le+1YM7j2sv6TArNRNnk70ETjJ7Yk0u wiDvrYujFQTdTRzd7E1SnEHqAMYKXaoTkyvHLzdDTnq4J7GFhwu8kNIvxtQnFpOK c+BLAbC7QhHiMgxDQYG7ROZUtvt15zrVWVOKwzXrVwFYFDTBKZvsSxSBYcbP1Jup Z7J8JkeDrn0RQshudQJKd+/Js2X6nNHrOkDbgC4K81HhgNWE5kUp/O3rO/pZaJHz a24qFx/Y9+zegAeGuZd/rUcDHJkEFa8oJoMDnhWwdTvt7XPVS7jprawk68h2AS1S /6HDJzu2lpkv8SppNJAGSgGnBB3RJrveCcCozb18brsYVZ4JLrsJ/j9x9Ot4PUMk uuYB5RvxutbmiBX0l6xa3Jox9gjKyjz1w7bNhDC2WOILWoIYkWcDbA6ETRJVqdvb IIkehSvSbTpb5BhD0kOZjbi1Pf6Btbb2JMmCs7ZhKRzGaedoqWU=
    =b8v+
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andrej Shadura@21:1/5 to Yaroslav Halchenko on Sat Jan 21 15:00:02 2023
    Hi,

    On Fri, 20 Jan 2023, at 17:01, Yaroslav Halchenko wrote:
    Dear Debian gurus,

    Lazy me would love to get automation going akin to the one in
    conda-forge (see e.g. [1]) -- whenever a new upstream is released
    (as could be detected with a run of uscan) the CI would

    <...>

    Is there any package/tool which already does something like that on CI?
    Or may be you see some issues with such a procedure?

    Not quite the same, but we have machinery to update packages from Debian in Apertis:
    https://gitlab.apertis.org/infrastructure/ci-package-builder/

    Related scripts live here: https://gitlab.apertis.org/infrastructure/apertis-docker-images/-/tree/apertis/v2024dev1/package-source-builder/overlay/usr/bin
    https://gitlab.apertis.org/pkg/apertis-dev-tools/-/tree/apertis/v2024dev1/tools

    --
    Cheers,
    Andrej

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Yaroslav Halchenko@21:1/5 to Andrej Shadura on Wed Jan 25 16:10:01 2023
    Dear Andrej, Paul and undef,

    I just wanted to say Thank you for all the pointers! Overall, janitor
    looks great and the most advanced and seems already covering various
    aspects but might be a bit "too big to tune for our own use-case", so we
    might indeed pick up on ideas/implementations... we will see.

    Cheers,

    On Sat, 21 Jan 2023, Andrej Shadura wrote:

    Hi,

    On Fri, 20 Jan 2023, at 17:01, Yaroslav Halchenko wrote:
    Dear Debian gurus,

    Lazy me would love to get automation going akin to the one in
    conda-forge (see e.g. [1]) -- whenever a new upstream is released
    (as could be detected with a run of uscan) the CI would

    <...>

    Is there any package/tool which already does something like that on CI?
    Or may be you see some issues with such a procedure?

    Not quite the same, but we have machinery to update packages from Debian in Apertis:
    https://gitlab.apertis.org/infrastructure/ci-package-builder/

    Related scripts live here: https://gitlab.apertis.org/infrastructure/apertis-docker-images/-/tree/apertis/v2024dev1/package-source-builder/overlay/usr/bin
    https://gitlab.apertis.org/pkg/apertis-dev-tools/-/tree/apertis/v2024dev1/tools
    --
    Yaroslav O. Halchenko
    Center for Open Neuroscience http://centerforopenneuroscience.org
    Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
    WWW: http://www.linkedin.com/in/yarik


    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCAAdFiEExbkF8OjZ/ZZo/zZvot4jUGLaM/oFAmPRRRwACgkQot4jUGLa M/pJNA/9GuFsxmrknPN07u/tK6VqRL3+OLA1lGBG6hcgTek3izq7GySCHR5T+Lkf 3BgAjsLR+1Ld5myBdm8mVxFF1pvVPkwyBum3JMGXLBzP3DBUlWhlkYX6/sp4Wzqo UNfQxYiykFgAxYA9uyrDfUG95IAYSQ5K+IjHwl6AnGTc3bINEtrNTiKt18uOQWf/ H5SUy/Y2c5r5AzqXLoXjNSRDAf26dgF5LlkRiSXVGSh2XT+CGUe2iK4WvhiB7nIb hFeTnDFvoaayr7kZW0+XIQJo7wBBDwAyQyWtz5LuuGJYbHaDaUAt5XrxUMu7Gnrv c3yYjTS7mh1HNwAnNrTXVS2f6C8r39oGBLwiGRhNYXwHPFDE/AO9wwTPeVuQjVXX d9filYlO6pUJRLpz6Tw26g5oWxuwxpDm4A1HnhX2mtIK08Wnzy2fu0bLRYszSA8V 4/fzyc2ito/PeGbSPd0+k8nAL2MdQwfBA9Q6u2zGo8yK41GTxJ5JNHrUUcZ5ld3i nb4GYC7VluMdWzLxpCExpUj1WG1KywkKFbUd4LPg9KqeqzjjM/Y8fgNasiwJIMro LY1M6YaMJGL7zdon9nj7aGfI9IRE3PuU2P2MGT+Ihzxm2yUdQ83eiomlrdMegEt5 0PO8BPXBWJouJD0iUKI7C4GRF3A9PIW9oohUno/dS/
  • From Jelmer =?utf-8?Q?Vernoo=C4=B3?=@21:1/5 to Yaroslav Halchenko on Wed Jan 25 17:00:01 2023
    Hi Yaroslav,

    On Wed, Jan 25, 2023 at 10:05:03AM -0500, Yaroslav Halchenko wrote:
    I just wanted to say Thank you for all the pointers! Overall, janitor
    looks great and the most advanced and seems already covering various
    aspects but might be a bit "too big to tune for our own use-case", so we might indeed pick up on ideas/implementations... we will see.

    The janitor runs the "deb-new-upstream" command under the hood to make the actual changes. You should be able to run that from e.g. CI. It tries to
    do the right thing as much as possible without flags, but I'm happy to
    add flags to allow tweaking its behaviour.

    Alternatively, if the package is in Debian you could simply merge
    the changes the Janitor has already published at a predictable
    location (https://janitor.debian.net/git/<source-pkg> -b fresh-releases)

    Cheers,

    Jelmer


    On Sat, 21 Jan 2023, Andrej Shadura wrote:

    Hi,

    On Fri, 20 Jan 2023, at 17:01, Yaroslav Halchenko wrote:
    Dear Debian gurus,

    Lazy me would love to get automation going akin to the one in
    conda-forge (see e.g. [1]) -- whenever a new upstream is released
    (as could be detected with a run of uscan) the CI would

    <...>

    Is there any package/tool which already does something like that on CI? Or may be you see some issues with such a procedure?

    Not quite the same, but we have machinery to update packages from Debian in Apertis:
    https://gitlab.apertis.org/infrastructure/ci-package-builder/

    Related scripts live here: https://gitlab.apertis.org/infrastructure/apertis-docker-images/-/tree/apertis/v2024dev1/package-source-builder/overlay/usr/bin
    https://gitlab.apertis.org/pkg/apertis-dev-tools/-/tree/apertis/v2024dev1/tools
    --
    Yaroslav O. Halchenko
    Center for Open Neuroscience http://centerforopenneuroscience.org Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
    WWW: http://www.linkedin.com/in/yarik


    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andreas Tille@21:1/5 to All on Fri Feb 17 21:30:01 2023
    Hi Yaroslav,

    Am Wed, Jan 25, 2023 at 10:05:03AM -0500 schrieb Yaroslav Halchenko:
    I just wanted to say Thank you for all the pointers! Overall, janitor
    looks great and the most advanced and seems already covering various
    aspects but might be a bit "too big to tune for our own use-case", so we might indeed pick up on ideas/implementations... we will see.

    May be a bit late, but may be

    apt install routine-update

    is your friend. It also calls scripts that are used by Janitor.

    Kind regards
    Andreas.

    --
    http://fam-tille.de

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jelmer =?utf-8?Q?Vernoo=C4=B3?=@21:1/5 to Andreas Tille on Fri Feb 17 23:50:01 2023
    On Fri, Feb 17, 2023 at 09:26:17PM +0100, Andreas Tille wrote:
    Hi Yaroslav,

    Am Wed, Jan 25, 2023 at 10:05:03AM -0500 schrieb Yaroslav Halchenko:
    I just wanted to say Thank you for all the pointers! Overall, janitor looks great and the most advanced and seems already covering various aspects but might be a bit "too big to tune for our own use-case", so we might indeed pick up on ideas/implementations... we will see.

    May be a bit late, but may be

    apt install routine-update

    is your friend. It also calls scripts that are used by Janitor.

    Does it call deb-new-upstream as well? That's the one that mainly seems relevant here.

    Jelmer

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andreas Tille@21:1/5 to All on Sat Feb 18 11:50:01 2023
    Am Fri, Feb 17, 2023 at 10:32:06PM +0000 schrieb Jelmer Vernooij:

    is your friend. It also calls scripts that are used by Janitor.

    Does it call deb-new-upstream as well? That's the one that mainly seems relevant here.

    I just learned about deb-new-upstream. Its parsing uscan manually but its not very robust and I should probably switch to use deb-new-upstream.

    Kind regards
    Andreas.

    --
    http://fam-tille.de

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andreas Tille@21:1/5 to All on Sat Feb 25 10:10:02 2023
    Am Sat, Feb 18, 2023 at 11:25:11AM +0100 schrieb Andreas Tille:
    Does it call deb-new-upstream as well? That's the one that mainly seems relevant here.

    I just learned about deb-new-upstream. Its parsing uscan manually but its not
    very robust and I should probably switch to use deb-new-upstream.

    I just learned that routine-update is currently broken due to a changed
    output by uscan (need to check this later when I have more time). This
    gave me sufficient motivation to check out deb-new-upstream. I admit
    that bug #1031913 is some kind of show stopper for me currently to
    consider it as a replacement for the current strategy in routine-update
    to simply use uscan.

    How to you work around this in janitor?

    Kind regards
    Andreas.

    --
    http://fam-tille.de

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)