Unpack crates in parallel using xargs to utilize multicore systems
better. Perform checksumming via a single sha256sum invocation.

For dev-python/watchfiles, this speeds up unpacking on my machine
from 2.6 s to 0.75 s (warm cache).

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
eclass/cargo.eclass | 56 ++++++++++++++++++++++++++-------------------
1 file changed, 33 insertions(+), 23 deletions(-)

diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
index 0f2da982f60c..5a16d3a30528 100644
--- a/eclass/cargo.eclass
+++ b/eclass/cargo.eclass
@@ -329,40 +329,50 @@ _cargo_gen_git_config() {
cargo_src_unpack() {
debug-print-function ${FUNCNAME} "$@"

- mkdir -p "${ECARGO_VENDOR}" || die
- mkdir -p "${S}" || die
+ mkdir -p "${ECARGO_VENDOR}" "${S}" || die

local archive shasum pkg
+ local crates=()
for archive in ${A}; do
case "${archive}" in
*.crate)
- # when called by pkgdiff-mg, do not unpack crates
- [[ ${PKGBUMPING} == ${PVR} ]] && continue
-
- ebegin "Loading ${archive} into Cargo registry" - tar -xf "${DISTDIR}"/${archive} -C "${ECARGO_VENDOR}/" || die
- # generate sha256sum of the crate itself as cargo needs this
- shasum=$(sha256sum "${DISTDIR}"/${archive} | cut -d ' ' -f 1)
- pkg=$(basename ${archive} .crate)

Michał Górny <mgorny@gentoo.org> writes:

Unpack crates in parallel using xargs to utilize multicore systems
better. Perform checksumming via a single sha256sum invocation.

For dev-python/watchfiles, this speeds up unpacking on my machine
from 2.6 s to 0.75 s (warm cache).

Signed-off-by: Michał Górny <mgorny@gentoo.org>

For completeness (acked on PR), lgtm. I assume tested by building
all cargo inheritees (nearly said 'heirs' but.. lol)?

Give a chance for others to look though.

Also, thank you!

---
eclass/cargo.eclass | 56 ++++++++++++++++++++++++++-------------------
1 file changed, 33 insertions(+), 23 deletions(-)

diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
index 0f2da982f60c..5a16d3a30528 100644
--- a/eclass/cargo.eclass
+++ b/eclass/cargo.eclass
@@ -329,40 +329,50 @@ _cargo_gen_git_config() {
cargo_src_unpack() {
debug-print-function ${FUNCNAME} "$@"

- mkdir -p "${ECARGO_VENDOR}" || die
- mkdir -p "${S}" || die
+ mkdir -p "${ECARGO_VENDOR}" "${S}" || die

local archive shasum pkg
+ local crates=()
for archive in ${A}; do
case "${archive}" in
*.crate)
- # when called by pkgdiff-mg, do not unpack crates
- [[ ${PKGBUMPING} == ${PVR} ]] && continue
-
- ebegin "Loading ${archive} into Cargo registry" - tar -xf "${DISTDIR}"/${archive} -C "${ECARGO_VENDOR}/" || die
- # generate sha256sum of the crate itself as cargo needs this
- shasum=$(sha256sum "${DISTDIR}"/${archive} | cut -d ' ' -f 1)
- pkg=$(basename ${archive} .crate)
- cat <<- EOF > ${ECARGO_VENDOR}/${pkg}/.cargo-checksum.json
- {
- "package": "${shasum}",
- "files": {}
- }
- EOF
- # if this is our target package we need it in ${WORKDIR} too
- # to make ${S} (and handle any revisions too)
- if [[ ${P} == ${pkg}* ]]; then
- tar -xf "${DISTDIR}"/${archive} -C "${WORKDIR}" || die
- fi
- eend $?
+ crates+=( "${archive}" )
;;
*)
- unpack ${archive}
+ unpack "${archive}"
;;
esac
done

+ if [[ ${PKGBUMPING} != ${PVR} ]]; then
+ pushd "${DISTDIR}" >/dev/null || die
+
+ ebegin "Unpacking crates"
+ printf '%s\0' "${crates[@]}" |
+ xargs -0 -P "$(makeopts_jobs)" -n 1 -- \
+ tar -x -C "${ECARGO_VENDOR}" -f
+ assert
+ eend $?
+
+ while read -d '' -r shasum archive; do
+ pkg=${archive%.crate}
+ cat <<- EOF > ${ECARGO_VENDOR}/${pkg}/.cargo-checksum.json || die
+ {
+ "package": "${shasum}",
+ "files": {}
+ }
+ EOF
+
+ # if this is our target package we need it in ${WORKDIR} too
+ # to make ${S} (and handle any revisions too)
+ if [[ ${P} == ${pkg}* ]]; then
+ tar -xf "${archive}" -C "${WORKDIR}" || die
+ fi
+ done < <(sha256sum -z "${crates[@]}" || die)
+
+ popd >/dev/null || die
+ fi
+
cargo_gen_config
}

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sun, 2024-05-12 at 19:22 +0200, Florian Schmaus wrote:

On 12/05/2024 04.26, Michał Górny wrote:

Unpack crates in parallel using xargs to utilize multicore systems
better. Perform checksumming via a single sha256sum invocation.

For dev-python/watchfiles, this speeds up unpacking on my machine
from 2.6 s to 0.75 s (warm cache).

Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
eclass/cargo.eclass | 56 ++++++++++++++++++++++++++-------------------
1 file changed, 33 insertions(+), 23 deletions(-)

diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
index 0f2da982f60c..5a16d3a30528 100644
--- a/eclass/cargo.eclass
+++ b/eclass/cargo.eclass
@@ -329,40 +329,50 @@ _cargo_gen_git_config() {
cargo_src_unpack() {
debug-print-function ${FUNCNAME} "$@"

- mkdir -p "${ECARGO_VENDOR}" || die
- mkdir -p "${S}" || die
+ mkdir -p "${ECARGO_VENDOR}" "${S}" || die

local archive shasum pkg
+ local crates=()
for archive in ${A}; do
case "${archive}" in
*.crate)
- # when called by pkgdiff-mg, do not unpack crates
- [[ ${PKGBUMPING} == ${PVR} ]] && continue
-
- ebegin "Loading ${archive} into Cargo registry" - tar -xf "${DISTDIR}"/${archive} -C "${ECARGO_VENDOR}/" || die
- # generate sha256sum of the crate itself as cargo needs this
- shasum=$(sha256sum "${DISTDIR}"/${archive} | cut -d ' ' -f 1)
- pkg=$(basename ${archive} .crate)
- cat <<- EOF > ${ECARGO_VENDOR}/${pkg}/.cargo-checksum.json
- {
- "package": "${shasum}",
- "files": {}
- }
- EOF
- # if this is our target package we need it in ${WORKDIR} too
- # to make ${S} (and handle any revisions too)
- if [[ ${P} == ${pkg}* ]]; then
- tar -xf "${DISTDIR}"/${archive} -C "${WORKDIR}" || die
- fi
- eend $?
+ crates+=( "${archive}" )
;;
*)
- unpack ${archive}
+ unpack "${archive}"
;;
esac
done

+ if [[ ${PKGBUMPING} != ${PVR} ]]; then
+ pushd "${DISTDIR}" >/dev/null || die
+
+ ebegin "Unpacking crates"
+ printf '%s\0' "${crates[@]}" |
+ xargs -0 -P "$(makeopts_jobs)" -n 1 -- \

Consider using get_makeopts_jobs instead of makeopts_jobs, as it
searches more variables for --jobs.

Whose bright idea was to add a second similarly named function that does roughly the same thing but apparently differently? It can hardly get
more confusing.

--
Best regards,
Michał Górny


-----BEGIN PGP SIGNATURE-----

iQFGBAABCgAwFiEEx2qEUJQJjSjMiybFY5ra4jKeJA4FAmZBCKcSHG1nb3JueUBn ZW50b28ub3JnAAoJEGOa2uIyniQOV9sIAL/g89ByXcB8Dt0+c3nV3gzGXBUyd6yy UobR+PWRv1qGM6qmdxOfBnkSMAjpbPk1jf3e0sO0q4a01EKcb3H/IOFQfeSwED2X +qO7oDcIzS7XRiIl7wEWX99qcE6+7N5QbLSI2Gle43oBGU7VjKGFghrK8Syy6REV PYS5lnEK4fvhvmz0nC97/58Xht8o1VocorclKcPqj7mUul3+jz8cMDcqoUMHGqJ/ Rkuw0dz+Ahk260aQWX7D6jEr7LUi+VumA3hg6yo+L3u+MJFMMkeD/rNTOOjzqW6N YrSS83exevM1qKPShbYLpHW6+Qef7LXB1+WVUef2aWcxNi85o3r6xRc=
=7haw
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 12/05/2024 20.21, Michał Górny wrote:

On Sun, 2024-05-12 at 19:22 +0200, Florian Schmaus wrote:

On 12/05/2024 04.26, Michał Górny wrote:

+ if [[ ${PKGBUMPING} != ${PVR} ]]; then
+ pushd "${DISTDIR}" >/dev/null || die
+
+ ebegin "Unpacking crates"
+ printf '%s\0' "${crates[@]}" |
+ xargs -0 -P "$(makeopts_jobs)" -n 1 -- \

Consider using get_makeopts_jobs instead of makeopts_jobs, as it
searches more variables for --jobs.

Whose bright idea was to add a second similarly named function that does roughly the same thing but apparently differently? It can hardly get
more confusing.

You are absolutely right, it sucks that we have two very similar methods.

You are invited to suggest how the situation can be improved. However,
rambling without presenting alternatives is not helpful in any way.

Potentially, you will either discover that there is a reason why things
are the way they are, or find a better solution.

- Flow

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, 2024-05-13 at 16:00 +0200, Florian Schmaus wrote:

On 12/05/2024 20.21, Michał Górny wrote:

On Sun, 2024-05-12 at 19:22 +0200, Florian Schmaus wrote:

On 12/05/2024 04.26, Michał Górny wrote:

+ if [[ ${PKGBUMPING} != ${PVR} ]]; then
+ pushd "${DISTDIR}" >/dev/null || die
+
+ ebegin "Unpacking crates"
+ printf '%s\0' "${crates[@]}" |
+ xargs -0 -P "$(makeopts_jobs)" -n 1 -- \

Consider using get_makeopts_jobs instead of makeopts_jobs, as it
searches more variables for --jobs.

Whose bright idea was to add a second similarly named function that does roughly the same thing but apparently differently? It can hardly get
more confusing.

You are absolutely right, it sucks that we have two very similar methods.

You are invited to suggest how the situation can be improved. However, rambling without presenting alternatives is not helpful in any way.

My suggestion would be for the person who introduced new methods
and implicitly claimed the old methods to be "legacy" to put an actual
effort to migrate consumers.

--
Best regards,
Michał Górny


-----BEGIN PGP SIGNATURE-----

iQFGBAABCgAwFiEEx2qEUJQJjSjMiybFY5ra4jKeJA4FAmZCHX8SHG1nb3JueUBn ZW50b28ub3JnAAoJEGOa2uIyniQOozAIALyLvLOQMUMrMnfqHX+3eZxo+5OjzRw8 1usAubpISZ0TXoxQL19BaYF2Z2cNvPO9eiHz770M1csJyvAxghSFCBwq2RhsC1nI b3Oql20hjZQc1gF+8ouavFg0F6TzmY9f0SMqmzhw6bkqKehIKySr7qRISTwsHUof RdYprSDVJPCaOOJI053Qq635sgcSsmLJqt1DftP/wOzIR0Zvvyyfg75KFVfvCTUX EhuC7trZg/VJhlXsfGK688e7HsBUS6t1uKcgvK+BY04t/SvOeIHtDDcJ8+xfGxcS r7HZh439CKIfu01+6ZlRSz70gOuS8ifUNIeJ+l7/LWRCYqUWWJHDMQg=
=Ld05
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)