Pop-Up Thingie

>>> Magnum BBS <<<
  • Home
  • Forum
  • Files
  • Log in

  1. Forum
  2. Usenet
  3. LINUX.GENTOO.DEV
  • [gentoo-dev] [PATCH] profiles/thirdpartymirrors: add final fallback for

    From Eli Schwartz@21:1/5 to All on Wed Oct 23 05:30:01 2024
    Apache has a mirror network, which only covers the most recent release
    of any given package. They also have an additional site which does not
    appear to be a CDN, and is throttled and can maybe ban you if you use it
    too much. Unfortunately, it is also the ONLY way to actually get
    historic releases of many packages.

    Use it, and use it last -- after every other mirror has been tried,
    which should handle latest releases. In combination with GENTOO_MIRRORS
    this should ensure that users can actually download software when
    needed, without running afoul of throttling.

    Signed-off-by: Eli Schwartz <eschwartz@gentoo.org>
    ---
    profiles/thirdpartymirrors | 2 +-
    1 file changed, 1 insertion(+), 1 deletion(-)

    diff --git a/profiles/thirdpartymirrors b/profiles/thirdpartymirrors
    index bb5398896976..d3a857327495 100644
    --- a/profiles/thirdpartymirrors
    +++ b/profiles/thirdpartymirrors
    @@ -1,4 +1,4 @@
    -apache https://dlcdn.apache.org/ https://apache.mirror.iphh.net/ https://artfiles.org/apache.org/ https://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.apache.org/dist/ https://ftp.fau.de/apache/ https://apache.osuosl.org/
    +apache https://dlcdn.apache.org/ https://apache.mirror.iphh.net/ https://artfiles.org/apache.org/ https://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.apache.org/dist/ https://ftp.fau.de/apache/ https://apache.osuosl.org/ https://archive.apache.org/dist/
    cpan https://cpan.metacpan.org https://www.cpan.org
    debian https://deb.debian.org/debian/ http://ftp.au.debian.org/debian/ http://ftp.at.debian.org/debian/ http://ftp.by.debian.org/debian/ http://ftp.be.debian.org/debian/ http://ftp.br.debian.org/debian/ ht
  • From Sam James@21:1/5 to Eli Schwartz on Fri Oct 25 18:20:02 2024
    Eli Schwartz <eschwartz@gentoo.org> writes:

    Apache has a mirror network, which only covers the most recent release
    of any given package. They also have an additional site which does not
    appear to be a CDN, and is throttled and can maybe ban you if you use it
    too much. Unfortunately, it is also the ONLY way to actually get
    historic releases of many packages.

    Use it, and use it last -- after every other mirror has been tried,
    which should handle latest releases. In combination with GENTOO_MIRRORS
    this should ensure that users can actually download software when
    needed, without running afoul of throttling.

    Signed-off-by: Eli Schwartz <eschwartz@gentoo.org>
    ---

    LGTM.

    profiles/thirdpartymirrors | 2 +-
    1 file changed, 1 insertion(+), 1 deletion(-)

    diff --git a/profiles/thirdpartymirrors b/profiles/thirdpartymirrors
    index bb5398896976..d3a857327495 100644
    --- a/profiles/thirdpartymirrors
    +++ b/profiles/thirdpartymirrors
    @@ -1,4 +1,4 @@
    -apache https://dlcdn.apache.org/ https://apache.mirror.iphh.net/ https://artfiles.org/apache.org/ https://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.apache.org/dist/ https://ftp.fau.de/apache/ https://apache.osuosl.org/
    +apache https://dlcdn.apache.org/ https://apache.mirror.iphh.net/ https://artfiles.org/apache.org/ https://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.apache.org/dist/ https://ftp.fau.de/apache/ https://apache.osuosl.org/ https://archive.apache.org/dist/
    cpan https://cpan.metacpan.org https://www.cpan.org
    debian https://deb.debian.org/debian/
    http://ftp.au.debian.org/debian/ http://ftp.at.debian.org/debian/ http://ftp.by.debian.org/debian/ http://ftp.be.debian.org/debian/ http://ftp.br.debian.org/debian/ http://ftp.bg.debian.org/debian/ http://ftp.ca.debian.org/debian/ http://ftp2.cn.debian.org/debian/ http://ftp.cn.debian.org/debian/ http://ftp.hr.debian.org/debian/ http://ftp.cz.debian.org/debian/ http://ftp.dk.debian.org/debian/ http://ftp.sv.debian.org/debian/ http://ftp.ee.debian.org/debian/ http://ftp.fi.debian.org/debian/ http://ftp.fr.debian.org/debian/ http://ftp2.de.debian.org/debian/ http://ftp.de.debian.org/debian/ http://ftp.gr.debian.org/debian/ http://ftp.hu.debian.org/debian/ http://ftp.is.debian.org/debian/ http://ftp.ie.debian.org/debian/ http://ftp.it.debian.org/debian/ http://ftp.jp.debian.org/debian/ http://ftp.lt.debian.org/debian/ http://ftp.mx.debian.org/debian/ http://ftp.md.debian.org/debian/ http://ftp.nl.debian.org/debian/ http://ftp.nc.debian.org/debian/ http://ftp.nz.debian.org/debian/ http://ftp.no.debian.org/debian/ http://ftp.pl.debian.org/debian/ http://ftp.pt.debian.org/debian/ http://ftp.ro.debian.org/debian/ http://ftp.ru.debian.org/debian/ http://ftp.sg.debian.org/debian/ http://ftp.sk.debian.org/debian/ http://ftp.si.debian.org/debian/ http://ftp.es.debian.org/debian/ http://ftp.se.debian.org/debian/ http://ftp.ch.debian.org/debian/ http://ftp.tw.debian.org/debian/ http://ftp.tr.debian.org/debian/ http://ftp.ua.debian.org/debian/ http://ftp.uk.debian.org/debian/ http://ftp.us.debian.org/debian/
    gentoo https://distfiles.gentoo.org/distfiles https://gentoo.osuosl.org/distfiles https://ftp.halifax.rwth-aachen.de/gentoo/distfiles https://ftp.fau.de/gentoo/distfiles

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Robin H. Johnson@21:1/5 to Eli Schwartz on Sat Oct 26 07:10:01 2024
    On Tue, Oct 22, 2024 at 11:24:29PM -0400, Eli Schwartz wrote:
    Apache has a mirror network, which only covers the most recent release
    of any given package. They also have an additional site which does not
    appear to be a CDN, and is throttled and can maybe ban you if you use it
    too much. Unfortunately, it is also the ONLY way to actually get
    historic releases of many packages.

    Use it, and use it last -- after every other mirror has been tried,
    which should handle latest releases. In combination with GENTOO_MIRRORS
    this should ensure that users can actually download software when
    needed, without running afoul of throttling.
    This does not actually do it "last" as you claim.

    Portage shuffles the list of thirdpartymirrors: https://gitweb.gentoo.org/proj/portage.git/tree/lib/portage/package/ebuild/fetch.py#n1140

    While it increases mirror burden; this should likely be done as a
    distinct thirdpartymirror:
    apache-historical https://archive.apache.org/dist/

    And that gets used in ebuilds when distfiles fall off the main mirrors
    [until such time as strictly ordered behavior is available].

    In likelihood, unless the ebuild has RESTRICT=mirror, I'd hope that that
    the primary request is made to Gentoo's mirrors before the upstreams: so
    that we are better open-source participants.

    Users who are also trying to use old ebuilds, where the distfile is no
    longer on the Gentoo mirrors should also consider thirdpartymirrors in
    their profiles, in which case inserting this entry locally would be appropriate.

    --
    Robin Hugh Johnson
    Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
    E-Mail : robbat2@gentoo.org
    GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
    GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v2
    Comment: Robbat2 @ Orbis-Terrarum Networks - The text below is a digital signature. If it doesn't make any sense to you, ignore it.

    iQKTBAABCgB9FiEEveu2pS8Vb98xaNkRGTlfI8WIJsQFAmcceBFfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEJE RUJCNkE1MkYxNTZGREYzMTY4RDkxMTE5Mzk1RjIzQzU4ODI2QzQACgkQGTlfI8WI JsTDARAAhpOmbZ+u3fVJQVhCjS8gIliZPqfeBbqB9uFWf3yzFv5nbCvjn4cFdhRF 9tOZBe3oUr+niqMiMg6lsWCW4pzR7ntMui6kpX7j85uYyEA9JcsoFc3TsEk8fx6e ce3yLcsmZB5DbRbmeuMamsLjScbxhXFOaAPbvfHqVPpea2Uj6XpcosoMSoJqymRJ Xxl39GDNuS/jbh2vMk3qTPS7WP3KJv/GfiOcdgq6om5kewVhDlkwHcirhSO4Wo/7 +uaHEsm449fOFMtMA33CoIcF2dri9/WrX4L/zr4G7QJfdVBTDRhv6HZ44d0/090l MakfAo3DWd/ni63O/m5frdtlSTcZoQtuh5QRHsvrcnyfUEB3O5HjgNdL0MmIayhw o98hZOW5IObYL/WGCZT1
  • From Mike Gilbert@21:1/5 to robbat2@gentoo.org on Sat Oct 26 17:30:01 2024
    On Sat, Oct 26, 2024 at 1:03 AM Robin H. Johnson <robbat2@gentoo.org> wrote:

    On Tue, Oct 22, 2024 at 11:24:29PM -0400, Eli Schwartz wrote:
    Apache has a mirror network, which only covers the most recent release
    of any given package. They also have an additional site which does not appear to be a CDN, and is throttled and can maybe ban you if you use it too much. Unfortunately, it is also the ONLY way to actually get
    historic releases of many packages.

    Use it, and use it last -- after every other mirror has been tried,
    which should handle latest releases. In combination with GENTOO_MIRRORS this should ensure that users can actually download software when
    needed, without running afoul of throttling.
    This does not actually do it "last" as you claim.

    Portage shuffles the list of thirdpartymirrors: https://gitweb.gentoo.org/proj/portage.git/tree/lib/portage/package/ebuild/fetch.py#n1140

    While it increases mirror burden; this should likely be done as a
    distinct thirdpartymirror:
    apache-historical https://archive.apache.org/dist/

    And that gets used in ebuilds when distfiles fall off the main mirrors
    [until such time as strictly ordered behavior is available].

    I assume Eli was trying to avoid having to update old ebuilds.

    I don't really see any point in adding an apache-historical entry in thirdpartymirrors; just inline https://archive.apache.org/dist/ into
    SRC_URI if the ebuild requires it.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Eli Schwartz@21:1/5 to Robin H. Johnson on Sun Oct 27 03:00:01 2024
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------iLY3J0nKRO0dbgKAGxYgT97t
    Content-Type: text/plain; charset=UTF-8
    Content-Transfer-Encoding: quoted-printable

    On 10/26/24 1:03 AM, Robin H. Johnson wrote:
    This does not actually do it "last" as you claim.

    Portage shuffles the list of thirdpartymirrors: https://gitweb.gentoo.org/proj/portage.git/tree/lib/portage/package/ebuild/fetch.py#n1140

    While it increases mirror burden; this should likely be done as a
    distinct thirdpartymirror:
    apache-historical https://archive.apache.org/dist/


    Thanks for pointing this out. It's disappointing but my opinion is that
    this changes nothing other than the commit message I should use when
    committing it.


    And that gets used in ebuilds when distfiles fall off the main mirrors
    [until such time as strictly ordered behavior is available].


    This is an unreasonable and irrational burden on ebuild maintainers, and regardless of your choice of remediation, if it isn't automatically and transparently handling it via the thirdpartymirrors handling itself,
    nobody wins. Not ebuild maintainers (who have to both know the quirks of
    apache and watch more closely than usual for updates), not bug triagers
    (who run the risk of getting trivially solvable issues to sort through),
    and not users (who run a supported configuration and end up with
    availability issues).

    It's a bug (albeit low priority) if SRC_URI doesn't point to a valid
    download location -- why should the fact that thirdpartymirrors has no
    priority selection change that fact?

    The suggestion downthread about extending this somehow in EAPI 9 is
    clever -- we should do it, but we also shouldn't leave the current state
    of affairs broken until some unspecified time years in the future.


    In likelihood, unless the ebuild has RESTRICT=mirror, I'd hope that that
    the primary request is made to Gentoo's mirrors before the upstreams: so
    that we are better open-source participants.


    Yes. This is why I think any change to thirdpartymirrors is low risk.


    Users who are also trying to use old ebuilds, where the distfile is no
    longer on the Gentoo mirrors should also consider thirdpartymirrors in
    their profiles, in which case inserting this entry locally would be appropriate.


    Unacceptable.

    This isn't about old ebuilds. It's about the very latest ebuilds.
    Distfiles go missing from the upstream CDN and are only available via
    long-term archival, the same minute (AFAIK, based on their
    documentation) that a new patchlevel release is tagged upstream. It is physically impossible for Gentoo to be that fast in packaging new
    versions, and it doesn't matter anyway because it won't help for another
    month due to stabilization. And that is assuming it gets packaged as
    fast as the maintainer can package it -- forgive me for being skeptical
    about that one, we are all human after all. :)

    It is reasonable, expected, normal use of computers, and reasonable,
    expected, normal use of Gentoo, to download and install "old" versions
    of a program minutes or hours after another patch release is out. We
    must not penalize those users by requiring them to manually implement
    this patch at global scope, running the risk that they will get it wrong
    and list only the slow site which they then overload for *all* traffic
    rather than at least having shuffle().

    For indeed, overriding a mirror in /etc/portage/mirrors is just going to
    force that to be used first for ALL packages using mirror://apache/ at
    all. At least it respects your listed ordering I guess...


    --
    Eli Schwartz

    --------------iLY3J0nKRO0dbgKAGxYgT97t--

    -----BEGIN PGP SIGNATURE-----

    wnsEABYIACMWIQTnFNnmK0TPZHnXm3qEp9ErcA0vVwUCZx2dmwUDAAAAAAAKCRCEp9ErcA0vVwB5 AQDHa3m9VCIkN+dqdz4mAVvRoCA/KcXlPKi7h+ZF3jpZpAEAuEGBhS6yyXKmuap7s9SuwYmDvHq2 IelZWgJcRaGn6wk=
    =5fwT
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • Who's Online

  • Recent Visitors

    • Guest
      Sun Jun 15 18:58:11 2025
      from No via SSH
    • Plume
      Sun Jun 15 15:01:03 2025
      from Uk via SSH
    • Centurion
      Sun Jun 15 09:44:59 2025
      from Berea, Ohio via Telnet
    • Deasl
      Sun Jun 15 08:43:59 2025
      from Foo, Bar via SSH
    • Deasl
      Sun Jun 15 08:41:06 2025
      from Foo, Bar via SSH
    • Plume
      Sat Jun 14 21:49:07 2025
      from Uk via SSH
    • Max Prime
      Sat Jun 14 16:47:41 2025
      from United Kingdom via SSH
    • Deasl
      Sat Jun 14 16:38:22 2025
      from Foo, Bar via SSH
  • System Info

    Sysop: Keyop
    Location: Huddersfield, West Yorkshire, UK
    Users: 493
    Nodes: 16 (3 / 13)
    Uptime: 174:26:47
    Calls: 9,705
    Calls today: 5
    Files: 13,736
    Messages: 6,178,746

© >>> Magnum BBS <<<, 2025