• [gentoo-user] sys-apps/portage and binary packages

    From Peter Humphrey@21:1/5 to All on Sat Jul 13 14:50:01 2024
    Hello list,

    Where I live, updates to portage itself usually take longer to appear as a binary package than as source, so I can't 'getbinpkg'. Therefore I've set:

    # cat /etc/portage/env/nobinpkg.conf
    FEATURES="${FEATURES} -getbinpkg"

    # cat /etc/portage/package.env
    sys-apps/portage nobinpkg.conf

    But still portage wants to fetch the binary.

    What am I doing wrong?

    --
    Regards,
    Peter.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Peter Humphrey@21:1/5 to All on Sat Jul 13 16:50:01 2024
    On Saturday, 13 July 2024 14:18:09 BST Arve Barsnes wrote:
    On Sat, 13 Jul 2024 at 14:42, Peter Humphrey <peter@prh.myzen.co.uk>
    wrote:
    Hello list,

    Where I live, updates to portage itself usually take longer to appear as a binary package than as source, so I can't 'getbinpkg'. Therefore I've set:

    # cat /etc/portage/env/nobinpkg.conf
    FEATURES="${FEATURES} -getbinpkg"

    # cat /etc/portage/package.env
    sys-apps/portage nobinpkg.conf

    But still portage wants to fetch the binary.

    What am I doing wrong?

    I don't know what you're doing wrong, but FEATURES is an additive
    variable, so adding the ${FEATURES} in there is not necessary.

    I've tried it without that parenthesis but with no difference.

    An alternative might be adding it to emerge default opts in make.conf:

    EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --usepkg-exclude 'sys-apps/portage'"

    Interesting. I hadn't seen that construction before; I'll try it.

    Thanks Arve.

    --
    Regards,
    Peter.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Peter Humphrey@21:1/5 to All on Sun Jul 14 02:20:01 2024
    On Saturday, 13 July 2024 15:49:00 BST Peter Humphrey wrote:
    On Saturday, 13 July 2024 14:18:09 BST Arve Barsnes wrote:
    8
    I don't know what you're doing wrong, but FEATURES is an additive
    variable, so adding the ${FEATURES} in there is not necessary.

    I've tried it without that parenthesis but with no difference.

    An alternative might be adding it to emerge default opts in make.conf:

    EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --usepkg-exclude 'sys-apps/portage'"

    Interesting. I hadn't seen that construction before; I'll try it.

    It works, but what's wrong with the way I tried it?

    --
    Regards,
    Peter.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Eli Schwartz@21:1/5 to Peter Humphrey on Sun Jul 14 08:10:01 2024
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------M1iYKYaR5OSM07bTPrvnLsvo
    Content-Type: text/plain; charset=UTF-8
    Content-Transfer-Encoding: quoted-printable

    On 7/13/24 8:42 AM, Peter Humphrey wrote:
    Hello list,

    Where I live, updates to portage itself usually take longer to appear as a binary package than as source, so I can't 'getbinpkg'. Therefore I've set:

    # cat /etc/portage/env/nobinpkg.conf
    FEATURES="${FEATURES} -getbinpkg"

    # cat /etc/portage/package.env
    sys-apps/portage nobinpkg.conf

    But still portage wants to fetch the binary.

    What am I doing wrong?


    As a matter of curiosity, why do you need to do any such thing at all?

    If there is no binary package available for portage yet, then portage
    will automatically build it from source instead, which is exactly what
    setting -getbinpkg for it would do. So why bother?

    The only thing that setting -getbinpkg could do is prevent you from
    using a binary on the off chance that it happens to appear faster for you.


    Note that independent of whether it's useful to exclude this one
    package, the functionality doesn't work. You cannot set per-package
    getbinpkg, this is tracked as https://bugs.gentoo.org/463964


    --
    Eli Schwartz


    --------------M1iYKYaR5OSM07bTPrvnLsvo--

    -----BEGIN PGP SIGNATURE-----

    wnsEABYIACMWIQTnFNnmK0TPZHnXm3qEp9ErcA0vVwUCZpNqkAUDAAAAAAAKCRCEp9ErcA0vVxIn AP0a8D4khGz2w8lLQRwT493F/E3l8tYqdPX7DuTAygKQegEA+AolO1yhAn/4ZveSdDjqe3MR1dnS s9JFo+VteUNLegg=
    =7jlq
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From netfab@21:1/5 to All on Sun Jul 14 10:50:02 2024
    Le 14/07/24 à 02:18, Peter Humphrey a tapoté :
    It works, but what's wrong with the way I tried it?

    Here is the explanation :

    https://bugs.gentoo.org/463964

    Currently, FEATURES=getbinpkg only works as a global setting.

    And from :

    https://forums.gentoo.org/viewtopic-t-1166963.html

    FEATURES are not fully supported in package.env as they can cause
    circular effects (e.g. getbinpkg may cause a different version to get selected, which can result in the package.env entry no longer being applicable).

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Peter Humphrey@21:1/5 to That might be so if portage behaved on Sun Jul 14 14:10:01 2024
    On Sunday, 14 July 2024 07:05:04 BST Eli Schwartz wrote:

    As a matter of curiosity, why do you need to do any such thing at all?

    If there is no binary package available for portage yet, then portage
    will automatically build it from source instead, which is exactly what setting -getbinpkg for it would do. So why bother?

    It doesn't do that here. It tries to fetch the binary and bombs out when it can't be found. Then I have to edit make.conf to update Gentoo, then put it back as it was for the rest of the system.

    The only thing that setting -getbinpkg could do is prevent you from
    using a binary on the off chance that it happens to appear faster for you.

    That might be so if portage behaved as you said above.

    Note that independent of whether it's useful to exclude this one
    package, the functionality doesn't work. You cannot set per-package getbinpkg, this is tracked as https://bugs.gentoo.org/463964

    Yes, I saw that from netfab's post.

    --
    Regards,
    Peter.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Wols Lists@21:1/5 to Peter Humphrey on Sun Jul 14 14:30:02 2024
    On 14/07/2024 13:04, Peter Humphrey wrote:
    It doesn't do that here. It tries to fetch the binary and bombs out when it can't be found. Then I have to edit make.conf to update Gentoo, then put it back as it was for the rest of the system.

    Do you have PORTAGE_DEFAULT_OPTIONS or whatever it's called set? That's
    caused me similar grief - it messed up my attempts to update my profile,
    it regularly messed up my virtualbox updates, etc etc.

    And if you have to edit various environment variables in make.conf, you
    know you can override them on the command line?

    PORTAGE_DEFAULT_OPTIONS="" emerge virtualbox-modules

    as I had to do ...

    Cheers,
    Wol

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Peter Humphrey@21:1/5 to All on Sun Jul 14 15:20:01 2024
    On Sunday, 14 July 2024 13:22:14 BST Wols Lists wrote:
    On 14/07/2024 13:04, Peter Humphrey wrote:
    It doesn't do that here. It tries to fetch the binary and bombs out when
    it can't be found. Then I have to edit make.conf to update Gentoo, then
    put it back as it was for the rest of the system.

    Do you have PORTAGE_DEFAULT_OPTIONS or whatever it's called set? That's caused me similar grief - it messed up my attempts to update my profile,
    it regularly messed up my virtualbox updates, etc etc.

    Yes, I have EMERGE_DEFAULT_OPTS="--jobs=8 --load-average=8 --autounmask=n -- keep-going --nospinner"

    Nothing contentious there, I'd have thought.

    Then I added this after Arve's advice, and after my problem arose:

    EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --usepkg-exclude 'sys-apps/ portage'"

    And if you have to edit various environment variables in make.conf, you
    know you can override them on the command line?

    PORTAGE_DEFAULT_OPTIONS="" emerge virtualbox-modules

    as I had to do ...

    Useful reminder; thanks.

    --
    Regards,
    Peter.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Wol@21:1/5 to Peter Humphrey on Sun Jul 14 16:10:01 2024
    On 14/07/2024 14:15, Peter Humphrey wrote:
    Yes, I have EMERGE_DEFAULT_OPTS="--jobs=8 --load-average=8 --autounmask=n -- keep-going --nospinner"

    Nothing contentious there, I'd have thought.

    I didn't think I had anything contentious - --once-only and that was
    about it. I think that was actually Dale's suggestion - anything I
    emerge will get cleaned away by my next depclean unless I actively force
    it into the world file.

    But a load of stuff just wouldn't --update or emerge as a result ...

    Cheers,
    Wol

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Eli Schwartz@21:1/5 to Peter Humphrey on Sun Jul 14 18:00:01 2024
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------EO40fnoz8sfPoy0r0gM6Z0eM
    Content-Type: text/plain; charset=UTF-8
    Content-Transfer-Encoding: quoted-printable

    On 7/14/24 8:04 AM, Peter Humphrey wrote:
    On Sunday, 14 July 2024 07:05:04 BST Eli Schwartz wrote:

    As a matter of curiosity, why do you need to do any such thing at all?

    If there is no binary package available for portage yet, then portage
    will automatically build it from source instead, which is exactly what
    setting -getbinpkg for it would do. So why bother?

    It doesn't do that here. It tries to fetch the binary and bombs out when it can't be found. Then I have to edit make.conf to update Gentoo, then put it back as it was for the rest of the system.


    This indicates that the Packages index on the binhost server doesn't
    correspond to the actual packages which are available; it is
    broadcasting availability of a new sys-apps/portage binpackage but that binpackage is not actually present.

    It has nothing to do with sys-apps/portage, since if there is a lack of synchronization here then the specific details of packages does not
    matter at all. It just happens, by coincidence, to trigger for you with
    the sys-apps/portage package.

    This is nominally speaking impossible to happen as it implies the
    binhost is simply broken altogether, which of course never happens in
    real life because we are all perfect, right?

    But actual broken binhosts aside, I'm guessing your real issue is using
    a DNS round robin load balancer or similar as your binhost url. You're
    getting delivered the Packages index from one server, but actual
    *.gpkg.tar requests are being serviced by a different load balancing
    machine, and the two aren't precisely in sync.

    There are some similar bugs being tracked:

    https://bugs.gentoo.org/464906
    https://bugs.gentoo.org/831657
    https://bugs.gentoo.org/890491
    https://bugs.gentoo.org/865845


    The default gentoobinhost shipped in stage3 tarballs is
    distfiles.gentoo.org, try changing it to a specific mirror and see if
    that helps?

    --
    Eli Schwartz


    --------------EO40fnoz8sfPoy0r0gM6Z0eM--

    -----BEGIN PGP SIGNATURE-----

    wnsEABYIACMWIQTnFNnmK0TPZHnXm3qEp9ErcA0vVwUCZpPz2wUDAAAAAAAKCRCEp9ErcA0vV9Ls AP9j30u1CAgCUgKuQHeOrBw8X2KbdXjWkjjpVEUDP3YQ3gD+MDl0wvgJrfsh/gq0yD1fdxokrj7f /WQ/cYRt/Opk4wg=
    =Jxwv
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Wols Lists@21:1/5 to Dale on Mon Jul 15 09:20:01 2024
    On 15/07/2024 07:22, Dale wrote:
    Is this about the -1 or --oneshot option?

    Yes.

    Once your system is stable it's a damn good idea :-)

    Cheers,
    Wol

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From J. Aho@21:1/5 to Peter Humphrey on Mon Jul 15 12:00:01 2024
    On 13/07/2024 14.50, Peter Humphrey wrote:
    Where I live, updates to portage itself usually take longer to appear as a binary package than as source, so I can't 'getbinpkg'. Therefore I've set:
    The main issue is that you aren't syncing portage towards the bin
    server, which makes things out of sync and those you will be building a
    lot of the packages instead of fetching the binary files when they are
    built. One way to come around this issue is that when you sync your
    portage, wait a day before you check what files to update, this way the
    binary packages should have been built, but sure it's a pita.

    The proper solution had been that you could sync portage against the
    binhost and that only if the binhost do have two portage that is always
    in par with the packages that has been built. I have my own experience
    of this issue from the time when I had multiple machines at home running
    Gentoo and I had a build environment that built packages for me. I had
    to setup two copies of portage on the build machine, one was the one it
    synced and built against, and then there was the portage version that
    was provided to the clients in the LAN (sadly this has a complication
    for the clients using git instead of rsync), this was only updated after
    a successful build. All the client synced portage against the binhost.
    If they didn't, then I had the issue as you describe and I had
    discussion about this on the gentoo irc channels, but as back then few
    people used binhosts so they didn't understand the issue and those
    portage don't support to do what you want to do, to filter out new
    ebuilds that don't have a binary package at the binhost, just wait
    another 20 years and then maybe.


    --

    //Aho

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Peter Humphrey@21:1/5 to All on Mon Jul 15 14:40:01 2024
    On Monday, 15 July 2024 10:54:37 BST J. Aho wrote:

    The main issue is that you aren't syncing portage towards the bin
    server, which makes things out of sync and those you will be building a
    lot of the packages instead of fetching the binary files when they are
    built. One way to come around this issue is that when you sync your
    portage, wait a day before you check what files to update, this way the binary packages should have been built, but sure it's a pita.

    You're right. I had set the sync-uri to mirror.bytemark.co.uk, which I got
    from the handbook. but I still had GENTOO_MIRRORS="http:// www.mirrorservice.org/sites/distfiles.gentoo.org/" from an older installation.

    I've now set the mirror to bytemark; let's see if that helps.

    Thanks.

    --
    Regards,
    Peter.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Michael@21:1/5 to All on Tue Jul 16 09:50:00 2024
    On Monday, 15 July 2024 13:36:19 BST Peter Humphrey wrote:
    On Monday, 15 July 2024 10:54:37 BST J. Aho wrote:
    The main issue is that you aren't syncing portage towards the bin
    server, which makes things out of sync and those you will be building a
    lot of the packages instead of fetching the binary files when they are built. One way to come around this issue is that when you sync your portage, wait a day before you check what files to update, this way the binary packages should have been built, but sure it's a pita.

    You're right. I had set the sync-uri to mirror.bytemark.co.uk, which I got from the handbook. but I still had GENTOO_MIRRORS="http:// www.mirrorservice.org/sites/distfiles.gentoo.org/" from an older installation.

    I've now set the mirror to bytemark; let's see if that helps.

    Thanks.

    I recall having some trouble with bytemark in the past. A package build time dependency portage required to complete an update was not available on
    bytemark and the update failed. I tried a day later and the same problem came up. A cursory look revealed this mirror to be out of date on a number of packages. Eventually I move to different mirrors and the problem disappeared.

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCAAdFiEEXqhvaVh2ERicA8Ceseqq9sKVZxkFAmaWNDgACgkQseqq9sKV Zxm6zBAAyx3JX8IeLD8fG4WpyMJVPQsWodxeSHlBv6h63vl1UmQDIzHaNJZp3CyR gvMP89GZQpW4ZttlH4sx8p3VXFhaoGpBfyWTyRBo3NFa+7C1bQ3CSgSP21qq+5I4 dpUCQHqjVjD9LrhSz5KI64KUaANrmb7hGtUi/kVceDsbue3AMSZ69aMVXZRO2xO5 BikDzjxGS1tFdXhsnmweo2sI9NWMXUIsKSQIqoDC0fTHjmP5DasSPNZZxb3l+7B+ cWl93U5JBtNdQZNiEHaUfQZvkU+9/zSdzrxfl7EHa//D1vb17RkdegYbKdw9VzHv tjjVfnun3npKzSB5MZGJJtN0+L0dwgHcOoxYBFztSmg1JG/SN7p38aRaTY7C+1Ef dryN8k/1j2prJgIe0ThKRJ06JJxljig+JfXPVOcJp2rOBCxE6JzFexbfgupR0NSe E8lh289N5tFbx6jJqQdtbfCIJ7N002VzvFFExh8Gh7FYWRiiIydmO9sCOySlMif1 e+//Q/fcSCu/zO2bDjd59731R/7kkBVQlH8GuroctxYRqS6kgIObayF1wB/vSzD6 21+rsfp0OGwZzi7RUWlPcs+uj3z99ajlVgwybf7o0Nj3cm78ER972Qj9wYkWkf09 ZDr904enByxWj9Pn1qf7FJp86k1H4I20SLf1qJTJLmFlhhSjUf0=
    =5LLD
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Peter Humphrey@21:1/5 to All on Tue Jul 16 12:40:01 2024
    On Tuesday, 16 July 2024 09:50:00 BST Michael wrote:

    I recall having some trouble with bytemark in the past.

    Now you mention it, so do I, but I've been using it for quite a while now with no problems.

    I suppose servers come and go...

    --
    Regards,
    Peter.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Eli Schwartz@21:1/5 to J. Aho on Tue Jul 16 17:00:01 2024
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------lwWbWOGbD8UdKWsnX68q66aC
    Content-Type: text/plain; charset=UTF-8
    Content-Transfer-Encoding: quoted-printable

    On 7/15/24 5:54 AM, J. Aho wrote:
    The main issue is that you aren't syncing portage towards the bin
    server, which makes things out of sync and those you will be building a
    lot of the packages instead of fetching the binary files when they are
    built.


    As noted earlier, it's more complicated than that. Portage can handle
    the case you describe just fine, in the sense that you will be building
    a lot of the packages instead of fetching the binary files.

    In which case, you don't need to set FEATURES="-getbinpkg" on a per
    package basis, since they will simply be built from source anyway.

    The problem comes when portage says there *is* a binary package
    available, because the index of packages says there is one, then it
    tries to download the binary file itself and receives a 404 error. That
    causes portage to crash since it doesn't expect the 404.


    If they didn't, then I had the issue as you describe and I had
    discussion about this on the gentoo irc channels, but as back then few
    people used binhosts so they didn't understand the issue and those
    portage don't support to do what you want to do, to filter out new
    ebuilds that don't have a binary package at the binhost, just wait
    another 20 years and then maybe.


    I understand where you are coming from :) because I refused to become a
    Gentoo user until there were official binhosts.

    There is as of 2024 a tracking issue for the bug you described, and with
    that public record of what to improve, some progress has been made to
    get it to work. I expect it to take significantly less than 20 years to
    deploy: the current version of portage stabilized for amd64 causes a
    binhost to expose the git commit for gentoo.git that it was built for,
    and the plan is that you should be able to have `emerge --sync` just
    sync to that revision as announced by the binhost.

    I will admit that it took us 5 months from time of reporting until
    portage made the progress it has so far, but I'm inclined to blame that
    on FOSS software being FOSS and people having other things to do with
    their time. Hopefully we'll only need to wait another couple of months
    for it to be full solved. :)

    Please do consider watching https://bugs.gentoo.org/924772 for further progress.


    --
    Eli Schwartz


    --------------lwWbWOGbD8UdKWsnX68q66aC--

    -----BEGIN PGP SIGNATURE-----

    wnsEABYIACMWIQTnFNnmK0TPZHnXm3qEp9ErcA0vVwUCZpaIvgUDAAAAAAAKCRCEp9ErcA0vV7ng AP9qT9mN4ViwJD3evl61qVuz3VIcMpGgWLntpEyAR2USBAEAxQU4hmJXBAs4nvM/EV0Yi5Oy620D I46Z3Px8u2pUMgQ=
    =lOnE
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)