1. Forum
  2. Usenet
  3. LINUX.DEBIAN.MAINT.PYTHON

  • python-sigstore / python-tuf: request for packaging help

    From Simon Josefsson@21:1/5 to All on Thu Dec 5 23:40:01 2024
    Hi

    I am new to python debian packaging, and I'm looking for guidance and
    review my packaging. I'm happy to team-maintain these packages if
    someone can add me to the salsa group.

    Right now I am working on python-sigstore and my packaging is here:

    https://salsa.debian.org/jas/sigstore-python/

    as you can see in the pipeline, it currently fails due to lacking tuf:

    https://salsa.debian.org/jas/sigstore-python/-/jobs/6706412

    It seems python-tuf has an old ITP/RFS report here:

    https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=934151 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=931178

    I started from scratch on python-tuf with the latest upstream version,
    instead of trying to understand the old packaging work that I couldn't
    get to work. Upstream evolved a lot. My packaging is here:

    https://salsa.debian.org/jas/python-tuf/

    The self-tests tries seems to open some files which fails and I suspect
    it is because srcdir != builddir reasons, or something similar, see
    errors here:

    https://salsa.debian.org/jas/python-tuf/-/jobs/6707693

    ==================================== ERRORS ====================================
    ______________ ERROR collecting tests/test_metadata_generation.py ______________
    tests/test_metadata_generation.py:10: in <module>
    from tests.generated_data.generate_md import generate_all_files tests/generated_data/generate_md.py:60: in <module>
    os.mkdir(OUT_DIR)
    E FileNotFoundError: [Errno 2] No such file or directory: 'generated_data/ed25519_metadata'

    How is this (seamingly common) problem solved normally? Do we copy test
    data files into the build directory somehow? Do we include them in the package? Do we patch hard-coded paths like this to make it work?

    If anyone can review my python-tuf and python-sigstore packages, that
    would be appreciated -- all nit-picks and style advice is most
    recommended, as I have not worked on python packaging before. Don't
    assume I understand any of the debian/* content so feel free to question
    some decision.

    If you want to build python-tuf locally, it depends on a more recent
    version of python3-securesystemslib and you can find it here:

    https://salsa.debian.org/jas/securesystemslib/ https://salsa.debian.org/jas/securesystemslib/-/pipelines/774721 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1089125

    Thanks,
    /Simon

    -----BEGIN PGP SIGNATURE-----

    iIoEARYIADIWIQSjzJyHC50xCrrUzy9RcisI/kdFogUCZ1IrnBQcc2ltb25Aam9z ZWZzc29uLm9yZwAKCRBRcisI/kdFonl5AP9407P5j2HjSg1tp72r7+8R0wpQYXJa zOGoZCw6NI2C2AEA/MznxTl3QTIrwm9IkHNZu4jzRhUDFsUmNfs3Eap0tgI=
    =BMi2
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andrey Rakhmatullin@21:1/5 to Simon Josefsson on Fri Dec 6 08:40:01 2024
    On Thu, Dec 05, 2024 at 11:39:24PM +0100, Simon Josefsson wrote:
    The self-tests tries seems to open some files which fails and I suspect
    it is because srcdir != builddir reasons, or something similar, see
    errors here:

    https://salsa.debian.org/jas/python-tuf/-/jobs/6707693

    ==================================== ERRORS ====================================
    ______________ ERROR collecting tests/test_metadata_generation.py ______________
    tests/test_metadata_generation.py:10: in <module>
    from tests.generated_data.generate_md import generate_all_files tests/generated_data/generate_md.py:60: in <module>
    os.mkdir(OUT_DIR)
    E FileNotFoundError: [Errno 2] No such file or directory: 'generated_data/ed25519_metadata'

    How is this (seamingly common) problem solved normally? Do we copy test
    data files into the build directory somehow? Do we include them in the package? Do we patch hard-coded paths like this to make it work?

    It's not common and test data files are already copied because they are
    under tests/. It's because of relative paths and I can reproduce this
    problem by running pytest in the upstream's git checkout. See also
    tox.ini:

    # TODO: Consider refactoring the tests to not require the aggregation script
    # being invoked from the `tests` directory.

    No idea if patching the path is enough or you need PYBUILD_BEFORE_TEST
    with `cd {dir}/tests` or something like that.

    --
    WBR, wRAR

    -----BEGIN PGP SIGNATURE-----

    iQJhBAABCgBLFiEEolIP6gqGcKZh3YxVM2L3AxpJkuEFAmdSqTItFIAAAAAAFQAP cGthLWFkZHJlc3NAZ251cGcub3Jnd3JhckBkZWJpYW4ub3JnAAoJEDNi9wMaSZLh KN4P/1jsjTR7U6OrW1JmvtPzohZDgGrR+KazXh+0xd4FpkmIp2t7E6EtFaBvgMTT WHfUzUoBDLMjrRfMro8O6VNR599OJBV/K2DY+BCKMh1jhFwzmFAZUna8EWqQ+V2T H5jikvINp2ni0Jd7rUwWzIFSAv65fiptEw1y5ySiKZBFMYnEe/zQyv3UMELm35vG AK4nw/cfPthp2lldmBDGZAQrTRvFMMJPCKLkWtYf0xeFnLx4+hkF6jLDI77s7NpC LydEkEWVS7I5SqgeFM/0Rzjdql6VuHjl6ihJD8xzP7VeUyB/VwKkpr7NQVjmLkEr hoR5R9hXuYmpBJsEQxwD79YrIIcIoo3mHtb9ZGFggwAwYc9PuZFYA6CuVocTm238 8gZeMgLti6A7iJd1KdJ5aqRNlWy6Eoi14H9kJj9l+ST4YBjgQudYg88LKPi0THMW 92G4JwVZlsr9HrdMTNYIW+EtHHW5ErXyJzQuQW4rzXzMutAXZAFRq2sfX6qQ7TdD dOGSyYSpgU/AMfzm37NphoZNfWbGo/ETYJFoJFYVqiCR1HXMdpOR0fkwfPYLeK1k S2jVPiFuqHCVz8DxeBGBRWrqCy9BdLDpVDzRJDyUmhxo/hw6H3IT5wAhyqme7Ixt Pu6xhFKk78Bs1W0hJ/IjuXzXSjG9Yx56WCDkEdMTWWTatE1l
    =mJp7
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Simon Josefsson@21:1/5 to Andrey Rakhmatullin on Fri Dec 6 17:00:02 2024
    Andrey Rakhmatullin <wrar@debian.org> writes:

    On Thu, Dec 05, 2024 at 11:39:24PM +0100, Simon Josefsson wrote:
    The self-tests tries seems to open some files which fails and I suspect
    it is because srcdir != builddir reasons, or something similar, see
    errors here:

    https://salsa.debian.org/jas/python-tuf/-/jobs/6707693

    ==================================== ERRORS ====================================
    ______________ ERROR collecting tests/test_metadata_generation.py ______________
    tests/test_metadata_generation.py:10: in <module>
    from tests.generated_data.generate_md import generate_all_files
    tests/generated_data/generate_md.py:60: in <module>
    os.mkdir(OUT_DIR)
    E FileNotFoundError: [Errno 2] No such file or directory: 'generated_data/ed25519_metadata'

    How is this (seamingly common) problem solved normally? Do we copy test
    data files into the build directory somehow? Do we include them in the
    package? Do we patch hard-coded paths like this to make it work?

    It's not common and test data files are already copied because they are
    under tests/. It's because of relative paths and I can reproduce this
    problem by running pytest in the upstream's git checkout.

    You are right, I reported it upstream:

    https://github.com/theupdateframework/python-tuf/issues/2745

    I disabled that self-check meanwhile, and got a bit further:

    collected 184 items
    tests/test_api.py .................................... [ 19%] tests/test_examples.py FFF [ 21%] tests/test_fetcher_ng.py FFFF.FFFF... [ 27%] tests/test_metadata_eq_.py .... [ 29%] tests/test_metadata_serialization.py ......................... [ 43%] tests/test_repository.py ........ [ 47%] tests/test_trusted_metadata_set.py .......................... [ 61%] tests/test_updater_consistent_snapshot.py ... [ 63%] tests/test_updater_delegation_graphs.py ...... [ 66%] tests/test_updater_fetch_target.py ..... [ 69%] tests/test_updater_key_rotations.py .. [ 70%] tests/test_updater_ng.py .FF.FF.F.FFFF [ 77%] tests/test_updater_top_level_update.py ................................. [ 95%] ... [ 97%] tests/test_updater_validation.py .. [ 98%] tests/test_utils.py F.. [100%] ...
    ================== 21 failed, 163 passed, 5 warnings in 5.39s ==================

    See full log:

    https://salsa.debian.org/jas/python-tuf/-/jobs/6710680/viewer

    Several failures seems related to not being able to find some test file,
    any ideas how to fix this further?

    I'll see if I can figure out how to disable only the failing tests,
    pending some better fix. It seems a large portion of the self-tests now
    work so I think this is in good enough shape for an upload to NEW.

    /Simon

    --=-=-Content-Type: application/pgp-signature; name="signature.asc"

    -----BEGIN PGP SIGNATURE-----

    iIoEARYIADIWIQSjzJyHC50xCrrUzy9RcisI/kdFogUCZ1MdLxQcc2ltb25Aam9z ZWZzc29uLm9yZwAKCRBRcisI/kdForiJAQCoUkLXAQGLcaRIkDXghi03mdBZ8vZD md4pyjqNw/EySQEA1Gd66hDfwBvbbr1rYZjCw6YAUtYtPR+Emjs33mYEHQw=KSnd
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Andrey Rakhmatullin@21:1/5 to Simon Josefsson on Fri Dec 6 17:20:01 2024
    On Fri, Dec 06, 2024 at 04:50:07PM +0100, Simon Josefsson wrote:
    See full log:

    https://salsa.debian.org/jas/python-tuf/-/jobs/6710680/viewer

    Several failures seems related to not being able to find some test file,
    any ideas how to fix this further?

    examples/manual_repo/basic_repo.py? You can copy the examples folder using debian/pybuild.testfiles. But it looks like most of the failures are network-related, you may need to disable the pybuild-set dummy proxies
    (not sure how) for these to work if they use only localhost networking.

    --
    WBR, wRAR

    -----BEGIN PGP SIGNATURE-----

    iQJhBAABCgBLFiEEolIP6gqGcKZh3YxVM2L3AxpJkuEFAmdTIvAtFIAAAAAAFQAP cGthLWFkZHJlc3NAZ251cGcub3Jnd3JhckBkZWJpYW4ub3JnAAoJEDNi9wMaSZLh HCYP/RJ7INHiqKPYq/fZ7RUECruQM/3HtAMNdx+KgYgmfUhQ+uFe0LNKzgFYoaBD FKPDzcmxBcFCf47L4yoqsxnGHXIPOgvarPyZKoyR0uIR22qmExhJZH2nZ2UDj0NZ umN5fEShzYj7Aft7tRIq5mgsZzkwQ4rWX+n1YCjJkqHqxYpBh8y3ujI4m2YBtFgl hz2VDM0nE1XhjVxYgkc6qffayu/lGfMhqgJMf2iphkhg7LyMbuO9zkC/lrbUr1RO MSWbHPLVKAVVVhGom6DE6XWYK9Gg1LYfA4gTrbwV4Gr36ESpfn5KowqtmxkaC7pq OX8sRakyCwzRpoKtOEanJI4XDZP04JfD6rArXuDzGfZcILhbYS5eOY727SWdkxiJ cawnjnbqVCo3KGeDA5GS+pE96AKfrQVcItYhc+O1ydoRLkG8KwioX3EXzmMQFAil rcvgN5ewQ+xRIyFmTFCvv6tpwE9BzjwHhtxlHRilVyUNVQwNHKjWv7kPmx3mUEWA xOqTJ8n8KqlMPOSHoytsM+TFn0MiP+7NNmr3NNYNMm0vStPvesY6hFEH5XXWBaxI j7R4KQ8pewq+ks/lJZ5V7XbXJ8kZ1uRmSX+jfwVVjBsLhM55SvFTKLzYL/xNFIFz tHtDUfXqS4i7lnUxLynxmJmSdtZJHVmS85e7HcCA9ziB/5PF
    =Sqnk
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)