Copy: otto@debian.org (Otto =?UTF-8?B?S2Vrw6Rsw6RpbmVu?=)

I am trying to enable Salsa CI for the qt6-webengine package.

The package takes a long time to build, longer than the default Salsa
CI runner timeout of 3 hours. I learned how to work around this with
the pyinstaller package by creating my own GitLab Runner running on my hardware, which allowed me to specify any length of job timeout.

But with qt6-webengine, I have run into another problem. Qt6-
webengine has a massive code base (500 GB tarball, 3 TB extracted).
This leads to 4.3 G of artifacts that try to upload to Salsa after
extracting the source. Which fails because it is larger than Salsa’s
750 MiB limit.

https://salsa.debian.org/qt-kde-team/qt6/qt6-webengine/-/jobs/7400053

My motivation in enabling Salsa CI for qt6-webengine relates to
efforts to provide better security support, which would benefit from
Salsa CI. Information can be found at:

https://salsa.debian.org/qt-kde-team/qt6/qt6-webengine/-/
merge_requests/8

I understand why Salsa needs a limit on the uploading of artifacts.
My question is if there is any way to work around this for qt6-
webengine. Is there some way I can store the artifacts on my system,
similar to how I can implement my own runner? Can I disable the
artifacts or cache them locally if other tests need them? I don’t
think we generally need the artifacts, just the output displayed for
each test.

--
Soren Stoutner
soren@debian.org
-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEJKVN2yNUZnlcqOI+wufLJ66wtgMFAmf1vukACgkQwufLJ66w tgNVsxAArNw4t9yKIepnTK/VkYvCkrkRkyJAj2MALhMYDp6qC6HHic++thFWBWWA 1K/R3jKj5/nNF+dUpXiEA4V2RVNRFyUc+llApSx3uwLxYY8+SrlrTf7n/ynabQHK L3mlf2JfGMXq5ArgbJOWFRTO9oHoFwvJCu00GcmxddZwpR7pAUeOwJ56L7UrEvM5 PmOrGKNemFXGG+q43jH+XNEzS7eRncEFAAugtff5RXHB8g9kjKeznX/85SCGERLn hte8kfWmVawqkVJuBT/0/NeUNhamHASpMYDSafPJ1jsZqFB/WFsWLkaA6hFTONl5 8XVazYDA87FPAwGUhRrmSpUr4x5aaA0UuYHkWKjrDtaZLJNKe6Wlr8HZdvdTcTz0 STEI1CB1ISbm1RB65VSxJ9G0JhL3HPiaob405LoLvLKNiI8i2Y2wVni7BNBOAG7p oGEhtVPZpZ7MOe84vrKJKRZN/W27hnCutgwwtf8jkras5Ew84I7jxxOQo6KGol+d IZ01S24YEYcu8k6CyCqTASBgY4HEyN2SIVEKpGZH8psdFfxYqDcN9mTcbQP1RGn8 3ilSnn0DTJI6eZRdY+L3E86pVz3TG0okpWFbXpr67QesiRPgWMERyee/dJD2Zxgr fmrfQztaScIUSnpTd9IM6k7cY/yi4SHyd2KEQKrrpBYn+0xonJI=
=ICQE
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Copy: otto@debian.org (Otto =?UTF-8?B?S2Vrw6Rsw6RpbmVu?=)

On Tuesday, April 8, 2025 5:27:21 PM Mountain Standard Time Soren
Stoutner wrote:

I understand why Salsa needs a limit on the uploading of artifacts.
My question is if there is any way to work around this for qt6-
webengine. Is there some way I can store the artifacts on my
system, similar to how I can implement my own runner? Can I
disable the artifacts or cache them locally if other tests need
them? I don’t think we generally need the artifacts, just the
output displayed for each test.

I have disabled Salsa CI for qt6-webengine for now, but if anyone has
any ideas in the future about how to handle large packages I would be
happy to re-enable it.

--
Soren Stoutner
soren@debian.org
-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEJKVN2yNUZnlcqOI+wufLJ66wtgMFAmf//YUACgkQwufLJ66w tgNlvw//fqy3IZO6TYbFmI0fkESv10eiQBa5SXayfc9fVygHeEbvvlZqhQdvtQqU dsZzDZkPWgBt6Ku+9f6z4rPa324BYh40BlMQBJBmdVgsUllcsGbRYqEw4y8hnXlh BQbLORj+790JIY6waOLhzcwXSzMJRw3JFATtAIwUNuQw1r6CjYsSjQ66+Z5oa9RI Ah2xrF2z5smHA3PkURYPV3b0HHZsskzjlbjz/4h7lRcFXiyIZRU/sE2X/lsjKMDD zoOrvmtq6EoJKD0Im9L7MMJiQhVMUwuiZfOPIRAWRSkYYgYIMnhU2DQzw/j1eWC7 d5RHy2BULlMyi3LKundL4YxZc1d64To+SiK6ZMu33/QF7Kll9K4v45JjOx39757s Tv6ZaSvWQtn8nwi+07M1R8Uo1tJvKrZWRU8hduLas5WBI+81RM8u0CvAulai9Sdy kS0dfCiR/MUHcenz5qIaoF2MtsHHgDqkXnMT+l7ngUT6EjL9NSP3TJ6ctHLoXG1h w/jUBQn/BSAtWzPn4oenx/ePYiA/m++fmMMCdgb1fVY5LGifB2r2Y14s5ZKEHAZd JjWp0w8KZXIm4XnzgYCHKB6xjnH/37uR7PD58O+6zw2uiHpZpha/qkp9aRKeR5SA Jzj5hngNofrondmz9NNjmmGjoISI0FMNtq2MnG5w3yPFOxSZML0=
=06Y9
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 09/04/2025 01:27, Soren Stoutner wrote:

I am trying to enable Salsa CI for the qt6-webengine package.

The package takes a long time to build, longer than the default Salsa
CI runner timeout of 3 hours. I learned how to work around this with
the pyinstaller package by creating my own GitLab Runner running on my hardware, which allowed me to specify any length of job timeout.

But with qt6-webengine, I have run into another problem. Qt6-
webengine has a massive code base (500 GB tarball, 3 TB extracted).
This leads to 4.3 G of artifacts that try to upload to Salsa after
extracting the source. Which fails because it is larger than Salsa’s
750 MiB limit.

The 750MiB is warning in the extract-source [1] recipe itself [2].
The failure is a timeout that comes later, likely due to the working dir
being 4.3 GB.

Setting the variable 'SALSA_CI_MAX_ARTIFACTS_SIZE' may remove the
warning for you, but it will probably still timeout.

At this massive size, you probably want to exclude the extracted source
from being uploaded as artifacts, so perhaps override the recipe and
exclude all paths except log file? [3]

I don't know too well how extending works in the recipes, so I would
play around with a new recipe 'qt6we-extract-source' and put all of 'provisioning-extract-source' in it, but remove the 'artifacts' being
uploaded due to this bit...

.provisioning-extract-source: &provisioning-extract-source

[...]

extends:
- .artifacts-default-expire <---- REMOVE ME

[...]

If that works, then I'd read up on extending and try to override 'artifacts-default-expire' on all recipes.


1. https://salsa.debian.org/qt-kde-team/qt6/qt6-webengine/-/jobs/7400127
2. https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml
3. https://docs.gitlab.com/ci/jobs/job_artifacts/#without-excluded-files

https://salsa.debian.org/qt-kde-team/qt6/qt6-webengine/-/jobs/7400053

My motivation in enabling Salsa CI for qt6-webengine relates to
efforts to provide better security support, which would benefit from
Salsa CI. Information can be found at:

https://salsa.debian.org/qt-kde-team/qt6/qt6-webengine/-/
merge_requests/8

I understand why Salsa needs a limit on the uploading of artifacts.
My question is if there is any way to work around this for qt6-
webengine. Is there some way I can store the artifacts on my system,
similar to how I can implement my own runner? Can I disable the
artifacts or cache them locally if other tests need them? I don’t
think we generally need the artifacts, just the output displayed for
each test.

--
Regards,
Ahmad

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Friday, April 18, 2025 6:35:54 AM Mountain Standard Time Ahmad
Khalifa wrote:

On 09/04/2025 01:27, Soren Stoutner wrote:

I am trying to enable Salsa CI for the qt6-webengine package.

The package takes a long time to build, longer than the default
Salsa CI runner timeout of 3 hours. I learned how to work around
this with the pyinstaller package by creating my own GitLab
Runner running on my hardware, which allowed me to specify any
length of job timeout.

But with qt6-webengine, I have run into another problem. Qt6-
webengine has a massive code base (500 GB tarball, 3 TB
extracted).
This leads to 4.3 G of artifacts that try to upload to Salsa after extracting the source. Which fails because it is larger than
Salsa’s 750 MiB limit.

The 750MiB is warning in the extract-source [1] recipe itself [2].
The failure is a timeout that comes later, likely due to the working
dir being 4.3 GB.

Setting the variable 'SALSA_CI_MAX_ARTIFACTS_SIZE' may remove the
warning for you, but it will probably still timeout.

At this massive size, you probably want to exclude the extracted
source from being uploaded as artifacts, so perhaps override the
recipe and exclude all paths except log file? [3]

I don't know too well how extending works in the recipes, so I would
play around with a new recipe 'qt6we-extract-source' and put all of 'provisioning-extract-source' in it, but remove the 'artifacts'
being uploaded due to this bit...

.provisioning-extract-source: &provisioning-extract-source

[...]

extends:
- .artifacts-default-expire <---- REMOVE ME

[...]

If that works, then I'd read up on extending and try to override 'artifacts-default-expire' on all recipes.

I might play around with that a bit. I think the builds reuse the
extracted source artifact. Perhaps what I do is rewrite the build to
be the full pipeline. So, for the build, it extracts the source and
runs the build.


--
Soren Stoutner
soren@debian.org
-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEJKVN2yNUZnlcqOI+wufLJ66wtgMFAmgC4LYACgkQwufLJ66w tgO3mg//RcmAISGa0G9w1xK6k+ETPVbbbWL60znrjmzUg9WKpTbSFPmkTZ6TsMz0 yZq2ARxRNgDpmHAkdhvPPEX3oBy/vRCwBET6dkcSx+HhkE56Ca/HonQD76wpKsUV mw+lZvEjqdsmywBy7U1PjiS9V6tYhi/dgtG4GRuMIE0us1LBHZhgP8dq2lZL1z+p rZOzZ5KtfPRxFcT6ENkVqMoKEYY+YLuju3rxj9d5TnSFrfJz0BidnKkuxRvp5nAC mdHtRNpyUDDjcMmECDcdXTjLHhKh5rW9IB31WnSGhsrmmlKB5UCQPv9sEywtjY2/ 9eNtCG+nqK/TkoHM1rrRE0HsBISd4tNZjcuoBLX3RchT/yGzeyNn1u5Sk0LLaNN3 VrE3SEPVXSd/PMnL8vsZCY3fYMX8BSM2AsLBaPk6MHMRaIsoVxuGPwEipOfO2Fb6 zh5to3gViY6k0kJn8dxEHAG6t0/Daq2TazQCSQ79Xrmdwp1rvHwN86Usr15F/sIH Taercmm6MaJE55/QWFbDh7JvYexitlLKb1tRb4RtYXgC5lZYOW2/kJrChFFf2tTs e4xGMeBAh8GgabRrSBHGfixMdgsdje4qyI7Yi0HgQzfCJxy98qP4ieP9ZaFZreok NNkyttyG46+key5G9FEEs5ynn+LO80qodogjRBW3xGS/Fnlsvic=
=yAeV
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)