• [gentoo-user] Portage load control

    From Peter Humphrey@21:1/5 to All on Sat May 6 14:00:02 2023
    Hello list,

    I still don't know how this works. I ran a test over the last two days, and the result does not accord with 'man make.conf' nor 'man 1 make'.

    First, 'man make.conf' does not state that --load-average, if set, will override --jobs, as it clearly does.

    Second, the two pages contribute actively to the confusion between the emerge jobs submitted in parallel by portage and the concurrent tasks that may be launched by each of those.

    The test:

    I ran 'emerge -e @world' with EMERGE_DEFAULT_OPTS="--jobs=10 --load-
    average=40 ...". It took 350m46s.

    Then I ran the same -e with --load-average=40, but no --jobs and no -j. That took 351m21s - 35 seconds longer! What's worse, the load average was
    controlled at about 72, not 40. I watched it for some time, and even though
    all three load averages were at 72-75, portage kept on starting more packages.

    As far as I could see, swap was not touched.

    The machine has 24 threads and 64GB RAM (not to mention plenty of swap), so
    how was the 72 figure arrived at?

    I still don't know how to control the number of simultaneous compilations, short of limiting them to one.

    --
    Regards,
    Peter.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jack@21:1/5 to Peter Humphrey on Sat May 6 20:20:01 2023
    On 2023.05.06 07:50, Peter Humphrey wrote:
    Hello list,

    I still don't know how this works. I ran a test over the last two
    days, and
    the result does not accord with 'man make.conf' nor 'man 1 make'.

    First, 'man make.conf' does not state that --load-average, if set,
    will
    override --jobs, as it clearly does.

    Second, the two pages contribute actively to the confusion between
    the emerge
    jobs submitted in parallel by portage and the concurrent tasks that
    may be
    launched by each of those.

    The test:

    I ran 'emerge -e @world' with EMERGE_DEFAULT_OPTS="--jobs=10 --load- average=40 ...". It took 350m46s.

    Then I ran the same -e with --load-average=40, but no --jobs and no
    -j. That
    took 351m21s - 35 seconds longer! What's worse, the load average was controlled at about 72, not 40. I watched it for some time, and even
    though
    all three load averages were at 72-75, portage kept on starting more packages.

    As far as I could see, swap was not touched.

    The machine has 24 threads and 64GB RAM (not to mention plenty of
    swap), so
    how was the 72 figure arrived at?

    I still don't know how to control the number of simultaneous
    compilations,
    short of limiting them to one.

    --
    Regards,
    Peter.
    Minor point - are you sure ccache isn't affecting your results?

    I hope I'm not preaching to the choir, and I have NOT reread the
    various man pages, but the different options you mention (and some you
    don't) apply to different parts of the process. Some tell emerge
    whether or not to start working on another package, but once it starts
    the process, it has no control over how busy the machine can get. Then
    there are those that get passed to make. I wouldn't think so, but are
    you possibly confusing the two? Lastly, I don't see that those that
    apply to make would have any effect on packages that use ninja instead,
    so that might also contribute to the issue..

    Separate question, only vaguely related: is there any easy way to tell
    what build tools (ninja vs make, gcc vs clang vs ?) were used for an installed package without actually looking into the ebuild? It's
    probably not relevant to your question about controlling the load, but
    I used to rebuild everything after installing a new version of gcc, and
    have since realized there are many packages that make no use of gcc, so
    the rebuild serves no point, and I miss rebuilding packages that use
    clang after an upgrade of it and related tools.

    Jack

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)