1. Forum
  2. Usenet
  3. LINUX.DEBIAN.DEVEL

  • Tool to build Debian packages not requiring root in containers ?

    From Charles Plessy@21:1/5 to All on Wed May 8 00:30:01 2024
    Hello everybody,

    I just re-suscribed :)

    At work I have access to a nice cluster with plenty of nodes rich of 128
    cores and 512 Gb RAM. The nodes do not run Debian but Singularity is
    available for virtualisation (https://en.wikipedia.org/wiki/Singularity_(software)).

    And in Debian I am part of transitions involving more than 100 packages
    every 6 months (r-api-bioc-*)…

    I want to leverage our cluster to automate as much of the rebuilds as I
    can, but could not find the right tool. I tried to run sbuild in a
    Singularity image and this failed. However, I do not need the whole
    power of engines like sbuild, as none of the packages involved require
    root priviledges to build.

    Do you have a suggestion for a tool can run in user mode in a container
    image having access to local storage on the host, and that given a
    Debian source control file will download the dependencies and build the
    package ?

    Have a nice day,

    Charles

    --
    Charles Plessy Nagahama, Yomitan, Okinawa, Japan
    Debian Med packaging team http://www.debian.org/devel/debian-med Tooting from work, https://fediscience.org/@charles_plessy Tooting from home, https://framapiaf.org/@charles_plessy

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?B?T3R0byBLZWvDpGzDpGluZW4=?@21:1/5 to Charles Plessy on Wed May 8 05:20:01 2024
    Hi!

    On Tue, 7 May 2024 at 15:27, Charles Plessy <plessy@debian.org> wrote:
    ..
    I want to leverage our cluster to automate as much of the rebuilds as I
    can, but could not find the right tool. I tried to run sbuild in a Singularity image and this failed. However, I do not need the whole
    power of engines like sbuild, as none of the packages involved require
    root priviledges to build.

    Do you have a suggestion for a tool can run in user mode in a container
    image having access to local storage on the host, and that given a
    Debian source control file will download the dependencies and build the package ?

    Can you give me an example of a package you want to build and what is
    the starting point, and I can tell you what command to issue to https://salsa.debian.org/otto/debcraft to achieve it?

    It supports running Podman in user mode (=no root permissions needed),
    it loop-mounts a local directory (local storage), creates clean build containers on the fly similar to sbuild but is much easier and faster
    to use.

    Example of how to build one of your packages with just pointing it at
    the source git repo:

    $ debcraft build https://salsa.debian.org/med-team/altree.git
    Building container 'debcraft-debian-sid' in '/tmp/tmp.brCZRhn2lL/debcraft-container' for downloader use
    mkdir: created directory '/tmp/tmp.brCZRhn2lL/debcraft-container'
    STEP 1/10: FROM debian:sid
    ...
    $ ls -1 debcraft-build-altree-1715137513.a8c999a+master altree_1.3.2-2_amd64.build
    altree_1.3.2-2_amd64.buildinfo
    altree_1.3.2-2_amd64.changes
    altree_1.3.2-2_amd64.deb
    altree-dbgsym_1.3.2-2_amd64.deb
    altree-examples_1.3.2-2_all.deb
    control.log
    filelist.log
    lintian.log

    First build is a bit slow as it needs to download all the dependencies
    and create a container, but the second run of 'debcraft build' inside
    the source directory will be very fast as all container cache is
    reused.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Charles Plessy@21:1/5 to All on Wed May 8 08:10:01 2024
    Le Tue, May 07, 2024 at 08:17:31PM -0700, Otto Kekäläinen a écrit :

    Can you give me an example of a package you want to build and what is
    the starting point, and I can tell you what command to issue to https://salsa.debian.org/otto/debcraft to achieve it?

    It supports running Podman in user mode (=no root permissions needed),

    Hi Otto,

    it looks really great!

    Do you think you can make it work with Singularity/Apptainer instead of
    Podman? Our cluster runs only singularity 3.5.2 (https://docs.sylabs.io/guides/3.5/user-guide/). Debian has version
    4.1.2 in the singularity-container package.

    The conversion of a Docker container to the Singularity format is
    simple, and Singularity already mounts most of the local storage to make
    it visible and writable from within the container.

    The typical packages that I want to build are the r-bioc-* collection. Together, they represent a dependency graph deep of a dozen of layers,
    which makes transitions work-intensive.

    With tools like debcraft I would like to prepare a set of updated
    packages for which I know that the CI tests pass, and that can be
    uploaded all together at the same time when I we get green light from
    the Release team. (And to rebuild all of them if in the meantime the
    contents of Unstable have changed significantly).

    Have a nice day,

    Charles

    --
    Charles Plessy Nagahama, Yomitan, Okinawa, Japan
    Debian Med packaging team http://www.debian.org/devel/debian-med Tooting from home https://framapiaf.org/@charles_plessy
    - You do not have my permission to use this email to train an AI -

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?B?T3R0byBLZWvDpGzDpGluZW4=?@21:1/5 to All on Wed May 8 17:10:01 2024
    Hi!


    ti 7. toukok. 2024 klo 23.01 Charles Plessy <plessy@debian.org> kirjoitti:

    Le Tue, May 07, 2024 at 08:17:31PM -0700, Otto Kekäläinen a écrit :

    Can you give me an example of a package you want to build and what is
    the starting point, and I can tell you what command to issue to https://salsa.debian.org/otto/debcraft to achieve it?

    It supports running Podman in user mode (=no root permissions needed),

    Hi Otto,

    it looks really great!

    Do you think you can make it work with Singularity/Apptainer instead of Podman? Our cluster runs only singularity 3.5.2 (https://docs.sylabs.io/guides/3.5/user-guide/). Debian has version
    4.1.2 in the singularity-container package.

    The conversion of a Docker container to the Singularity format is
    simple, and Singularity already mounts most of the local storage to make
    it visible and writable from within the container.


    I read the docs on how Singularity is able to pull Docker images of Debian
    Sid and build on top of them, and run and exec just like Docker/Podman. Unfortunately it has its own Containerfile format ( https://docs.sylabs.io/guides/3.5/user-guide/quick_start.html#singularity-definition-files)
    and the commands have their own syntax. I guess Debcraft could be extended
    to support it, but that would require at least one Singularity user as
    frequent contributor to test and develop Singularity-compatibility.

    The entire code base is shell code. Perhaps you want to take a look if it
    looks hackable for you?

    <div dir="auto"><div>Hi!</div><div dir="auto"><br><br><div class="gmail_quote" dir="auto"><div dir="ltr" class="gmail_attr">ti 7. toukok. 2024 klo 23.01 Charles Plessy &lt;<a href="mailto:plessy@debian.org" target="_blank" rel="noreferrer">plessy@debian.
    org</a>&gt; kirjoitti:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Le Tue, May 07, 2024 at 08:17:31PM -0700, Otto Kekäläinen a écrit :<br>
    &gt; <br>
    &gt; Can you give me an example of a package you want to build and what is<br> &gt; the starting point, and I can tell you what command to issue to<br>
    &gt; <a href="https://salsa.debian.org/otto/debcraft" rel="noreferrer noreferrer noreferrer" target="_blank">https://salsa.debian.org/otto/debcraft</a> to achieve it?<br>
    &gt; <br>
    &gt; It supports running Podman in user mode (=no root permissions needed),<br>

    Hi Otto,<br>

    it looks really great!<br>

    Do you think you can make it work with Singularity/Apptainer instead of<br> Podman?  Our cluster runs only singularity 3.5.2<br>
    (<a href="https://docs.sylabs.io/guides/3.5/user-guide/" rel="noreferrer noreferrer noreferrer" target="_blank">https://docs.sylabs.io/guides/3.5/user-guide/</a>).  Debian has version<br>
    4.1.2 in the singularity-container package.<br>

    The conversion of a Docker container to the Singularity format is<br>
    simple, and Singularity already mounts most of the local storage to make<br>
    it visible and writable from within the container.<br></blockquote></div></div><div dir="auto"><br></div><div dir="auto">I read the docs on how Singularity is able to pull Docker images of Debian Sid and build on top of them, and run and exec just like
    Docker/Podman. Unfortunately it has its own Containerfile format (<a href="https://docs.sylabs.io/guides/3.5/user-guide/quick_start.html#singularity-definition-files">https://docs.sylabs.io/guides/3.5/user-guide/quick_start.html#singularity-definition-
    files</a>) and the commands have their own syntax. I guess Debcraft could be extended to support it, but that would require at least one Singularity user as frequent contributor to test and develop Singularity-compatibility.</div><div dir="auto"><br></
    <div dir="auto">The entire code base is shell code. Perhaps you want to take a look if it looks hackable for you?</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-
    left:1px #ccc solid;padding-left:1ex">
    </blockquote></div></div></div>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Charles Plessy@21:1/5 to All on Thu May 9 01:40:01 2024
    Le Wed, May 08, 2024 at 08:02:41AM -0700, Otto Kekäläinen a écrit :

    I read the docs on how Singularity is able to pull Docker images of Debian Sid and build on top of them, and run and exec just like Docker/Podman. Unfortunately it has its own Containerfile format ( https://docs.sylabs.io/guides/3.5/user-guide/quick_start.html#singularity-definition-files)
    and the commands have their own syntax. I guess Debcraft could be extended
    to support it, but that would require at least one Singularity user as frequent contributor to test and develop Singularity-compatibility.

    The entire code base is shell code. Perhaps you want to take a look if it looks hackable for you?

    Hi Otto,

    I looked at the code, and while it would be easy to replace the podman
    commands to run containers, I wonder if there isn't a major roadblock:

    The main use of Singularity containers is to provide static images for software. The default is that the image is read-only and has write
    access to the host filesystems. Thus, running apt upgrade in a
    singularity container isn't something that is done usually. It might
    even be impossible, although I am not expert enough to make that
    statement firmly.

    Is there a chance debcraft can work from a static container provided by
    the user?

    I think that the key problem I have is that I want to use a build Debian packages that need no root access and that do not need to install
    dependencies that need root access, and I want to do that with user
    privileges only.

    Have a nice day,

    Charles

    --
    Charles Plessy Nagahama, Yomitan, Okinawa, Japan
    Debian Med packaging team http://www.debian.org/devel/debian-med Tooting from home https://framapiaf.org/@charles_plessy
    - You do not have my permission to use this email to train an AI -

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Timo =?utf-8?Q?R=C3=B6hling?=@21:1/5 to All on Thu May 9 14:20:01 2024
    Hi Charles,

    * Charles Plessy <plessy@debian.org> [2024-05-08 07:27]:
    I want to leverage our cluster to automate as much of the rebuilds
    as I
    can, but could not find the right tool. I tried to run sbuild in a >Singularity image and this failed. However, I do not need the whole
    power of engines like sbuild, as none of the packages involved require
    root priviledges to build.
    Have you tried the unshare backend for sbuild? It uses Linux
    namespaces instead of full-blown root privileges, and works really
    great for my regular packaging work. I have not tried running it
    inside a virtualization container, though.


    Cheers
    Timo

    --
    ⢀⣴⠾⠻⢶⣦⠀ ╭────────────────────────────────────────────────────╮
    ⣾â â¢ â ’⠀⣿⡠│ Timo Röhling │
    ⢿⡄⠘⠷⠚⠋⠀ │ 9B03 EBB9 8300 DF97 C2B1 23BF CC8C 6BDD 1403 F4CA │
    ⠈⠳⣄⠀⠀⠀⠀ ╰────────────────────────────────────────────────────╯

    -----BEGIN PGP SIGNATURE-----

    iQIzBAEBCgAdFiEEmwPruYMA35fCsSO/zIxr3RQD9MoFAmY8vlcACgkQzIxr3RQD 9MqX/w/5Ad1h3+NBPIi5FIe/EMq0phChzhoA6L20PNgFElU/KGL5YTjo49zGOdXt BSCv21vo7YQpvcK/f5od45fP7OFRI08332aCtdcKwrX5Vd2P7hW4Biris4slaH8M jfhf/U2KyJktCtdAUHLUeiMVTvX17oYEqA5qjzJmb+iZwbhi8gY2XJNWZx9O84kq vmExXB2b5df/X9LrPZh6Yh7jjN7lrEXzEgR1jI1mLdUSSiYTy0IYZEm9hwmdPibr Uww8rjuul01paRxiOcZAVJ2+7/lNtkSrfh6oYvUW5U6vPQt+om2Z54KdycKGeZpy 074HwYRwCNecoybbuhYpZ2dLZXHi8wC1swuJ5kk3wiY