• Bug#1101363: dbcsr: FTBFS: tests timeout in some cases

    From Santiago Vila@21:1/5 to All on Wed Mar 26 15:30:01 2025
    Package: src:dbcsr
    Version: 2.8.0-2
    Severity: important
    Tags: ftbfs trixie sid

    Dear team:

    This package used to FTBFS on single-cpu systems, so I did a team upload
    and added this line to debian/rules:

    export PRTE_MCA_rmaps_default_mapping_policy=:oversubscribe

    which is completely standard among scientific packages dealing with OpenMPI.


    Now the package finally builds ok on the AWS instances with a single-CPU
    on which I tried, but paradoxically, now it fails with timeout on
    AWS instances with 2 CPUs, as in the log below.

    So, I request the help from other team members to find a solution which works everywhere.

    Note: Before the change, the package used to take 3 minutes and a half to build.
    Now it takes 40 minutes to build. Should I just extend the timeout?

    --------------------------------------------------------------------------------
    [...]
    debian/rules clean
    dh clean --buildsystem=cmake
    dh_auto_clean -O--buildsystem=cmake
    dh_autoreconf_clean -O--buildsystem=cmake
    dh_clean -O--buildsystem=cmake
    debian/rules binary
    dh binary --buildsystem=cmake
    dh_update_autotools_config -O--buildsystem=cmake
    dh_autoreconf -O--buildsystem=cmake
    dh_auto_configure -O--buildsystem=cmake
    cd obj-x86_64-linux-gnu && DEB_PYTHON_INSTALL_LAYOUT=deb PKG_CONFIG=/usr/bin/pkg-config cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=None -DCMAKE_INSTALL_SYSCONFDIR=/etc -DCMAKE_INSTALL_LOCALSTATEDIR=/var -DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON -
    DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON -DFETCHCONTENT_FULLY_DISCONNECTED=ON "-GUnix Makefiles" -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_INSTALL_LIBDIR=lib/x86_64-linux-gnu ..
    -- The C compiler identification is GNU 14.2.0
    -- The CXX compiler identification is GNU 14.2.0
    -- Detecting C compiler ABI info

    [... snipped ...]

    18: -------------------------------------------------------------------------------
    18: # multiplications 1
    18: max memory usage/rank 278.441984E+06
    18: # max total images/rank 2
    18: # max 3D layers 1
    18: # MPI messages exchanged 2
    18: MPI messages size (bytes):
    18: total size 1.808000E+03
    18: min size 840.000000E+00
    18: max size 968.000000E+00
    18: average size 904.000000E+00
    18: MPI breakdown and total messages size (bytes):
    18: size <= 128 0 0 18: 128 < size <= 8192 2 1808 18: 8192 < size <= 32768 0 0 18: 32768 < size <= 131072 0 0 18: 131072 < size <= 4194304 0 0 18: 4194304 < size <= 16777216 0 0 18: 16777216 < size 0 0 18: -------------------------------------------------------------------------------
    18:
    18: *** WARNING in dbcsr_mm.F:301 :: Using a non-square number of MPI ranks ***
    18: *** might lead to poor performance. Used ranks: 2 Suggested: 1 4 ***
    18:
    18: -------------------------------------------------------------------------------
    18: - -
    18: - DBCSR MESSAGE PASSING PERFORMANCE -
    18: - -
    18: -------------------------------------------------------------------------------
    18: ROUTINE CALLS AVE VOLUME [Bytes]
    18: MP_Bcast 19 12.
    18: MP_Allreduce 59 26.
    18: MP_Alltoall 2 48.
    18: MP_ISend 6 315.
    18: MP_IRecv 6 287.
    18: -------------------------------------------------------------------------------
    18/19 Test #18: dbcsr_test ............................................ Passed 1.00 sec
    test 19
    Start 19: dbcsr_tensor_test

    19: Test command: /usr/bin/mpiexec "-n" "2" "./dbcsr_tensor_test"
    19: Working Directory: /<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/tests
    19: Environment variables:
    19: OMP_NUM_THREADS=2
    19: Test timeout computed to be: 1500
    19: [c7a-large-1742981929][[11101,1],0][../../../../../../opal/mca/btl/tcp/btl_tcp_proc.c:266:mca_btl_tcp_proc_create_interface_graph] Unable to find reachable pairing between local and remote interfaces
    19: [c7a-large-1742981929][[11101,1],1][../../../../../../opal/mca/btl/tcp/btl_tcp_proc.c:266:mca_btl_tcp_proc_create_interface_graph] Unable to find reachable pairing between local and remote interfaces
    19: pgrid3-dimensions:
    19: 2 1 1
    19:
    19: pgrid4-dimensions:
    19: 2 1 1 1
    19:
    19: dist11:
    19: 0 1 0 1
    19:
    19: dist12:
    19: 0 0 0 0 0 0 0 0 0 0 0
    19:
    19: dist13:
    19: 0 0 0 0 0 0 0 0 0
    19:
    19: dist21:
    19: 0 1 0 1
    19:
    19: dist22:
    19: 0 0 0 0 0 0 0 0 0 0 0
    19:
    19: dist23:
    19: 0 0 0 0 0
    19:
    19: dist24:
    19: 0 0 0
    19:
    19: dist31:
    19: 0 1 0 1 0 1 0 1 0
    19:
    19: dist32:
    19: 0 0 0 0 0
    19:
    19: dist33:
    19: 0 0 0
    19:
    19: Creating dist objects...
    19:
    19: Creating tensors...
    19: Tensor 1
    19:
    19: Filling Tensor...
    19: Dimension: 3
    19: Reserving blocks...
    19: Tensor 2
    19:
    19: Filling Tensor...
    19: Dimension: 4
    19: Reserving blocks...
    19: Tensor 3
    19:
    19: Filling Tensor...
    19: Dimension: 3
    19: Reserving blocks...
    19: Contracting...
    19: --------------------------------------------------------------------------------
    19: DBCSR TENSOR CONTRACTION: (13|2) x (54|21) = (3|45)
    19: --------------------------------------------------------------------------------
    19: GLOBAL INFO OF (13|2)
    19: block dimensions: 4 11 9
    19: full dimensions: 25 83 74
    19: process grid dimensions: 2 1 1
    19: Block sizes:
    19: Dim 1: 3 9 12 1
    19: Dim 2: 4 2 3 1 9 2 32 10 5 8 7
    19: Dim 3: 7 3 8 7 9 5 10 23 2
    19: Block distribution:
    19: Dim 1: 0 1 0 1
    19: Dim 2: 0 0 0 0 0 0 0 0 0 0 0
    19: Dim 3: 0 0 0 0 0 0 0 0 0
    19: DISTRIBUTION OF (13|2)
    19: Number of non-zero blocks: 32
    19: Percentage of non-zero blocks: 8.08
    19: Average number of blocks per CPU: 16
    19: Maximum number of blocks per CPU: 20
    19: Average number of matrix elements per CPU: 8155
    19: Maximum number of matrix elements per CPU: 9756
    19: GLOBAL INFO OF (54|21)
    19: block dimensions: 4 11 5 3
    19: full dimensions: 25 83 32 28
    19: process grid dimensions: 2 1 1 1
    19: Block sizes:
    19: Dim 1: 3 9 12 1
    19: Dim 2: 4 2 3 1 9 2 32 10 5 8 7
    19: Dim 3: 8 1 4 13 6
    19: Dim 4: 4 2 22
    19: Block distribution:
    19: Dim 1: 0 1 0 1
    19: Dim 2: 0 0 0 0 0 0 0 0 0 0 0
    19: Dim 3: 0 0 0 0 0
    19: Dim 4: 0 0 0
    19: DISTRIBUTION OF (54|21)
    19: Number of non-zero blocks: 36
    19: Percentage of non-zero blocks: 5.45
    19: Average number of blocks per CPU: 18
    19: Maximum number of blocks per CPU: 21
    19: Average number of matrix elements per CPU: 74745
    19: Maximum number of matrix elements per CPU: 132456
    19: INDEX INFO
    19: tensor index: (bca) x (bcde) = (ade)
    19: matrix index: (ba|c) x (ed|cb) = (a|de)
    19: aligning tensor index with data
    19: INDEX INFO
    19: tensor index: (bac) x (edcb) = (ade)
    19: matrix index: (ba|c) x (ed|cb) = (a|de)
    19: large tensors: 1, 2; small tensor: 3
    19: sorting contraction indices
    19: compatibility of (13|2): Not compatible
    19: compatibility of (54|21): Transposed
    19: No redistribution of (54|21)
    19: Redistribution of (13|2) compatible with (54|21)
    19: compatibility of (13|2): Transposed
    19: compatibility of (3|45): Not compatible
    19: Redistribution of (3|45)
    19: compatibility of (3|45): Normal
    19: INDEX INFO
    19: tensor index: (bac) x (edcb) = (ade)
    19: matrix index: (a|cb) x (ed|cb) = (a|ed)
    19: GLOBAL INFO OF (13|2)
    19: block dimensions: 4 9 11
    19: full dimensions: 25 74 83
    19: process grid dimensions: 2 1 1
    19: Block sizes:
    19: Dim 1: 3 9 12 1
    19: Dim 2: 7 3 8 7 9 5 10 23 2
    19: Dim 3: 4 2 3 1 9 2 32 10 5 8 7
    19: Block distribution:
    19: Dim 1: 0 1 0 1
    19: Dim 2: 0 0 0 0 0 0 0 0 0
    19: Dim 3: 0 0 0 0 0 0 0 0 0 0 0
    19: DISTRIBUTION OF (13|2)
    19: Number of non-zero blocks: 32
    19: Percentage of non-zero blocks: 8.08
    19: Average number of blocks per CPU: 16
    19: Maximum number of blocks per CPU: 20
    19: Average number of matrix elements per CPU: 8155
    19: Maximum number of matrix elements per CPU: 9756
    19: --------------------------------------------------------------------------------
    19: DBCSR TAS MATRIX MULTIPLICATION: (13|2) matrix x (54|21) matrix = (3|45) matrix
    19: --------------------------------------------------------------------------------
    19: mm dims: 9 44 15
    19: MM PARAMETERS
    19: Est. number of matrix elements per CPU of result matrix: 5269
    19: Est. optimal split factor: 2
    19: No redistribution of (13|2) matrix and (54|21) matrix
    19: Change split factor of (13|2) matrix : No
    19: Change split factor of (54|21) matrix : No
    19: mm case: -- x --T = +
    19: SPLIT / PARALLELIZATION INFO
    19: splitting columns by factor 2
    19: global grid sizes: 1x 2
    19: grid sizes on subgroups: 1x 1
    19: GLOBAL INFO OF (13|2) matrix
    19: block dimensions: 9 44
    19: full dimensions: 74 2075
    19: process grid dimensions: 1 2
    19: Block sizes:
    19: Row: 7 3 8 7 9 5 10 23 2
    19: Column: 12 6 9 3 27 6 96 30 15 24 21 36 18 27 9 81 18 288 90 45 72 63 48 24 36 12 108 24 384 120 60 96 84 4 2 3 1 9 2 32 10 5 8 7
    19: Block distribution:
    19: Row: 0 0 0 0 0 0 0 0 0
    19: Column: 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1
    19: GLOBAL INFO OF (54|21) matrix
    19: block dimensions: 15 44
    19: full dimensions: 896 2075
    19: process grid dimensions: 1 2
    19: Block sizes:
    19: Row: 32 16 176 4 2 22 16 8 88 52 26 286 24 12 132
    19: Column: 12 6 9 3 27 6 96 30 15 24 21 36 18 27 9 81 18 288 90 45 72 63 48 24 36 12 108 24 384 120 60 96 84 4 2 3 1 9 2 32 10 5 8 7
    19: Block distribution:
    19: Row: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    19: Column: 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1
    19: GLOBAL INFO OF (3|45) matrix
    19: block dimensions: 9 15
    19: full dimensions: 74 896
    19: process grid dimensions: 1 2
    19: Block sizes:
    19: Row: 7 3 8 7 9 5 10 23 2
    19: Column: 32 16 176 4 2 22 16 8 88 52 26 286 24 12 132
    19: Block distribution:
    19: Row: 0 0 0 0 0 0 0 0 0
    19: Column: 0 1 0 0 1 1 0 1 1 0 1 1 0 0 0
    19: Change process grid: No
    19: DISTRIBUTION OF (13|2) matrix
    19: Number of non-zero blocks: 32
    19: Percentage of non-zero blocks: 8.08
    19: Average number of blocks per group: 16
    19: Maximum number of blocks per group: 20
    19: Average number of matrix elements per group: 8155
    19: Maximum number of matrix elements per group: 9756
    19: Average number of blocks per CPU: 16
    19: Maximum number of blocks per CPU: 20
    19: Average number of matrix elements per CPU: 8155
    19: Maximum number of matrix elements per CPU: 9756
    19: DISTRIBUTION OF (54|21) matrix
    19: Number of non-zero blocks: 36
    19: Percentage of non-zero blocks: 5.45
    19: Average number of blocks per group: 18
    19: Maximum number of blocks per group: 21
    19: Average number of matrix elements per group: 74745
    19: Maximum number of matrix elements per group: 132456
    19: Average number of blocks per CPU: 18
    19: Maximum number of blocks per CPU: 21
    19: Average number of matrix elements per CPU: 74745
    19: Maximum number of matrix elements per CPU: 132456
    19: DISTRIBUTION OF (3|45) matrix replicated
    19: Number of non-zero blocks: 23
    19: Percentage of non-zero blocks: 8.52
    19: Average number of blocks per group: 12
    19: Maximum number of blocks per group: 13
    19: Average number of matrix elements per group: 5351
    19: Maximum number of matrix elements per group: 7306
    19: Average number of blocks per CPU: 12
    19: Maximum number of blocks per CPU: 13
    19: Average number of matrix elements per CPU: 5351
    19: Maximum number of matrix elements per CPU: 7306
    19: Row distribution on subgroup:
    19: 0 0 0 0 0 0 0 0 0
    19: Column distribution on subgroup:
    19: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 19: MM PARAMETERS
    19: Number of matrix elements per CPU of result matrix: 3653
    19: Optimal split factor: 2
    19: --------------------------------------------------------------------------------
    19: TAS MATRIX MULTIPLICATION DONE
    19: --------------------------------------------------------------------------------
    19: GLOBAL INFO OF (3|45)
    19: block dimensions: 9 5 3
    19: full dimensions: 74 32 28
    19: process grid dimensions: 2 1 1
    19: Block sizes:
    19: Dim 1: 7 3 8 7 9 5 10 23 2
    19: Dim 2: 8 1 4 13 6
    19: Dim 3: 4 2 22
    19: Block distribution:
    19: Dim 1: 0 0 0 1 0 1 0 1 1
    19: Dim 2: 0 0 0 0 0
    19: Dim 3: 0 0 0
    19: DISTRIBUTION OF (3|45)
    19: Number of non-zero blocks: 21
    19: Percentage of non-zero blocks: 15.56
    19: Average number of blocks per CPU: 11
    19: Maximum number of blocks per CPU: 13
    19: Average number of matrix elements per CPU: 5269
    19: Maximum number of matrix elements per CPU: 9052
    19: GLOBAL INFO OF (3|45)
    19: block dimensions: 9 5 3
    19: full dimensions: 74 32 28
    19: process grid dimensions: 2 1 1
    19: Block sizes:
    19: Dim 1: 7 3 8 7 9 5 10 23 2
    19: Dim 2: 8 1 4 13 6
    19: Dim 3: 4 2 22
    19: Block distribution:
    19: Dim 1: 0 1 0 1 0 1 0 1 0
    19: Dim 2: 0 0 0 0 0
    19: Dim 3: 0 0 0
    19: DISTRIBUTION OF (3|45)
    19: Number of non-zero blocks: 29
    19: Percentage of non-zero blocks: 21.48
    19: Average number of blocks per CPU: 15
    19: Maximum number of blocks per CPU: 15
    19: Average number of matrix elements per CPU: 6431
    19: Maximum number of matrix elements per CPU: 9322
    19: --------------------------------------------------------------------------------
    19: TENSOR CONTRACTION DONE
    19: --------------------------------------------------------------------------------
    19: Testing get_info for Tensor 1...
    19: Name: (13|2)
    19: Data_type: 3
    19: ======= Process: 0 ========
    19: Total number of blocks:
    19: 4 11 9
    19:
    19: Total number of elements:
    19: 25 83 74
    19:
    19: Total number of local blocks:
    19: 2 11 9
    19:
    19: Total number of local elements:
    19: 15 83 74
    19:
    19: Pgrid dimensions:
    19: 2 1 1
    19:
    19: Process coordinates:
    19: 0 0 0
    19:
    19: blks_local:
    19: 0 2
    19:
    19: 0 1 2 3 4 5 6 7 8 9 10
    19:
    19: 0 1 2 3 4 5 6 7 8
    19:
    19: proc_dist:
    19: 0 1 0 1
    19:
    19: 0 0 0 0 0 0 0 0 0 0 0
    19:
    19: 0 0 0 0 0 0 0 0 0
    19:
    19: blk_size:
    19: 3 9 12 1
    19:
    19: 4 2 3 1 9 2 32 10 5 8 7
    19:
    19: 7 3 8 7 9 5 10 23 2
    19:
    19: blk_offset:
    19: 0 3 12 24
    19:
    19: 0 4 6 9 10 19 21 53 63 68 76
    19:
    19: 0 7 10 18 25 34 39 49 72
    19:
    19: ======= Process: 1 ========
    19: Total number of blocks:
    19: 4 11 9
    19:
    19: Total number of elements:
    19: 25 83 74
    19:
    19: Total number of local blocks:
    19: 2 11 9
    19:
    19: Total number of local elements:
    19: 10 83 74
    19:
    19: Pgrid dimensions:
    19: 2 1 1
    19:
    19: Process coordinates:
    19: 1 0 0
    19:
    19: blks_local:
    19: 1 3
    19:
    19: 0 1 2 3 4 5 6 7 8 9 10
    19:
    19: 0 1 2 3 4 5 6 7 8
    19:
    19: proc_dist:
    19: 0 1 0 1
    19:
    19: 0 0 0 0 0 0 0 0 0 0 0
    19:
    19: 0 0 0 0 0 0 0 0 0
    19:
    19: blk_size:
    19: 3 9 12 1
    19:
    19: 4 2 3 1 9 2 32 10 5 8 7
    19:
    19: 7 3 8 7 9 5 10 23 2
    19:
    19: blk_offset:
    19: 0 3 12 24
    19:
    19: 0 4 6 9 10 19 21 53 63 68 76
    19:
    19: 0 7 10 18 25 34 39 49 72
    19:
    19: Testing get_mapping_info for Tensor 1...
    19: ndim_nd = 3
    19: ndim1_2d = 2
    19: ndim2_2d = 1
    19: dims_2d_i8: 36 11
    19:
    19: dims_2d: 36 11
    19:
    19: dims_nd:
    19: 3 10 8
    19:
    19: dims1_2d:
    19: 3 8
    19:
    19: dims2_2d:
    19: 10
    19:
    19: map1_2d:
    19: 0 2
    19:
    19: map2_2d:
    19: 1
    19:
    19: map_nd:
    19: 0 2 1
    19:
    19: Base: 1
    19: col_major 1
    19:
    19: Testing c_dbcsr_t_contract_index...
    19:
    19: Result Indices on Rank 0
    19: 0 3 1
    19: 1 0 0
    19: 1 3 2
    19: 2 1 0
    19: 2 3 1
    19: 4 1 0
    19: 4 2 1
    19: 4 3 1
    19: 4 3 2
    19: 6 0 0
    19: 6 3 1
    19: 6 3 2
    19: 6 4 2
    19: Result Indices on Rank 1
    19: 3 2 1
    19: 5 0 0
    19: 5 2 0
    19: 5 2 1
    19: 7 1 0
    19: 7 1 2
    19: 7 2 0
    19: 7 2 1
    19/19 Test #19: dbcsr_tensor_test ..................................... Passed 2.55 sec

    95% tests passed, 1 tests failed out of 19

    Total Test time (real) = 2334.56 sec

    The following tests FAILED:
    11 - dbcsr_unittest1 (Timeout)
    Errors while running CTest
    make[2]: *** [Makefile:74: test] Error 8
    make[2]: Leaving directory '/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu' dh_auto_test: error: cd obj-x86_64-linux-gnu && make -j1 test ARGS\+=--verbose ARGS\+=-j1 returned exit code 2
    make[1]: *** [debian/rules:21: override_dh_auto_test] Error 25
    make[1]: Leaving directory '/<<PKGBUILDDIR>>'
    make: *** [debian/rules:12: binary] Error 2
    dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2 --------------------------------------------------------------------------------

    The above is just how the build ends and not necessarily the most relevant part.
    If required, the full build log is available here:

    https://people.debian.org/~sanvila/build-logs/202503/

    About the archive rebuild: The build was made on virtual machines from AWS, using sbuild and a reduced chroot with only build-essential packages.

    If you could not reproduce the bug please contact me privately, as I
    am willing to provide ssh access to a virtual machine where the bug is
    fully reproducible.

    If this is really a bug in one of the build-depends, please use
    reassign and add an affects on src:dbcsr, so that this is still
    visible in the BTS web page for this package.

    Thanks.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Santiago Vila@21:1/5 to All on Thu Mar 27 01:10:01 2025
    Hi. I found the way to increase the timeout, no help is requested really.

    Thanks.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Santiago Vila@21:1/5 to All on Thu Mar 27 12:10:01 2025
    forwarded 1101363 https://github.com/cp2k/dbcsr/issues/902
    thanks

    I said:
    I found the way to increase the timeout, no help is requested really.

    Well, it's not so simple, as it does not make sense that I have to
    use a timeout of 2 hours (or even more) when it used to take less
    than 4 minutes to build.

    So I have forwarded the issue to the URL above.

    Thanks.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)