Hi Chris,

Is this repeatable or perhaps a flaky test?

Cheers,

Jelmer

On Sat, Apr 19, 2025 at 12:42:27PM +0200, Chris Hofstaedtler wrote:

Source: breezy
Version: 3.3.11-1
Severity: serious
Tags: ftbfs
Justification: ftbfs
X-Debbugs-Cc: debian-s390@lists.debian.org
User: debian-s390@lists.debian.org
Usertags: s390

breezy FTBFS on the s390x buildds:

https://buildd.debian.org/status/logs.php?pkg=breezy&ver=3.3.11-1&arch=s390x >https://buildd.debian.org/status/fetch.php?pkg=breezy&arch=s390x&ver=3.3.11-1&stamp=1744775171&raw=0

======================================================================
ERROR: breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree(SFTPTransport,SFTPAbsoluteServer)
---------------------------------------------------------------------- >testtools.testresult.real._StringException: log: {{{
793.512 creating repository in file:///tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree%28SFTPTransport%2CSFTPAbsoluteServer%29/work/branch/.bzr/.
793.513 creating branch <breezy.bzr.branch.BzrBranchFormat7 object at 0x3ff17a48b90> in file:///tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree%28SFTPTransport%

2CSFTPAbsoluteServer%29/work/branch/

793.516 trying to create missing lock '/tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree(SFTPTransport,SFTPAbsoluteServer)/work/branch/.bzr/checkout/dirstate'
793.516 opening working tree '/tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree(SFTPTransport,SFTPAbsoluteServer)/work/branch'
793.520 preparing to commit
INFO Committing to: /tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree(SFTPTransport,SFTPAbsoluteServer)/work/branch/
793.521 Selecting files for commit with filter None
INFO Committed revision 1.
793.524 Committed revid b'jrandom@example.com-20250416020608-wtigrtiemex6q4g4' as revno 1.
793.529 preparing to commit
INFO Committing to: /tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree(SFTPTransport,SFTPAbsoluteServer)/work/branch/
793.530 Selecting files for commit with filter None
INFO added hello
INFO Committed revision 2.
793.534 Committed revid b'jrandom@example.com-20250416020608-yk262zf8uf070i3p' as revno 2.
793.535 run brz: ['push', 'sftp://foo:bar@127.0.0.1:48505/tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree%28SFTPTransport%2CSFTPAbsoluteServer%29/work/remote_branch', '--

directory', 'branch']

793.535 breezy version: 3.3.11
793.535 brz arguments: ['push', 'sftp://foo:bar@127.0.0.1:48505/tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree%28SFTPTransport%2CSFTPAbsoluteServer%29/work/remote_branch', '--

directory', 'branch']

793.537 opening working tree '/tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree(SFTPTransport,SFTPAbsoluteServer)/work/branch'
793.540 opening working tree '/tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree(SFTPTransport,SFTPAbsoluteServer)/work/branch'
793.547 creating repository in sftp://foo@127.0.0.1:48505/tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree%28SFTPTransport%2CSFTPAbsoluteServer%29/work/remote_branch/.bzr/.
793.552 Using fetch logic to copy between CHKInventoryRepository('file:///tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree%28SFTPTransport%2CSFTPAbsoluteServer%29/work/branch/.

bzr/repository/')(RepositoryFormat2a()) and CHKInventoryRepository('sftp://foo@127.0.0.1:48505/tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree%28SFTPTransport%
2CSFTPAbsoluteServer%29/work/remote_branch/.bzr/repository/')(RepositoryFormat2a())

793.552 fetching: <PendingAncestryResult heads:frozenset({b'jrandom@example.com-20250416020608-yk262zf8uf070i3p'}) repo:CHKInventoryRepository('file:///tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_

upload_without_working_tree%28SFTPTransport%2CSFTPAbsoluteServer%29/work/branch/.bzr/repository/')>

793.562 creating branch <breezy.bzr.branch.BzrBranchFormat7 object at 0x3ff17cfcb90> in sftp://foo@127.0.0.1:48505/tmp/testbzr-odoocew3.tmp/breezy.plugins.upload.tests.test_upload.TestUploadFromRemoteBranch.test_upload_without_working_tree%

28SFTPTransport%2CSFTPAbsoluteServer%29/work/remote_branch/

INFO Created new branch.
793.574 errors:
'Created new branch.\n'
}}}

Traceback (most recent call last):
File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/chk_map.py", line 182, in _read_bytes
return _get_cache()[key]
~~~~~~~~~~~~^^^^^
File "/build/reproducible-path/breezy-3.3.11/breezy/lru_cache.py", line 61, in __getitem__
node = cache[key]
~~~~~^^^^^
KeyError: (b'sha1:e1e466803f6dd86ee55fe44890ac4a470bb44f07',)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/groupcompress.py", line 1525, in _get_blocks
yield read_memo, cached[read_memo]
~~~~~~^^^^^^^^^^^
KeyError: (<breezy.bzr.btree_index.BTreeGraphIndex object at 0x3ff17ace890>, 926, 131)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/build/reproducible-path/breezy-3.3.11/breezy/plugins/upload/tests/test_upload.py", line 790, in test_upload_without_working_tree
self.do_full_upload(directory=self.remote_branch_url)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/build/reproducible-path/breezy-3.3.11/breezy/plugins/upload/tests/test_upload.py", line 208, in do_full_upload
upload.run(up_url, *args, **kwargs)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "/build/reproducible-path/breezy-3.3.11/breezy/commands.py", line 816, in run
return class_run(*args, **kwargs)
File "/build/reproducible-path/breezy-3.3.11/breezy/plugins/upload/cmds.py", line 587, in run
uploader.upload_full_tree()
~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/build/reproducible-path/breezy-3.3.11/breezy/plugins/upload/cmds.py", line 324, in upload_full_tree
for relpath, ie in self.tree.iter_entries_by_dir():
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/inventorytree.py", line 323, in iter_entries
for p, e in inv.iter_entries_by_dir(
~~~~~~~~~~~~~~~~~~~~~~~^
specific_file_ids=inventory_file_ids
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
):
^
File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/inventory.py", line 805, in iter_entries_by_dir
self._preload_cache()
~~~~~~~~~~~~~~~~~~~^^
File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/inventory.py", line 2195, in _preload_cache
pid_items = self.parent_id_basename_to_file_id.iteritems()
File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/chk_map.py", line 566, in iteritems
self._ensure_root()
~~~~~~~~~~~~~~~~~^^
File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/chk_map.py", line 162, in _ensure_root
self._root_node = self._get_node(self._root_node)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/chk_map.py", line 175, in _get_node
bytes = self._read_bytes(node)
File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/chk_map.py", line 185, in _read_bytes
bytes = next(stream).get_bytes_as("fulltext")
~~~~^^^^^^^^
File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/groupcompress.py", line 1570, in get_record_stream
for content_factory in self._get_remaining_record_stream(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
keys, orig_keys, ordering, include_delta_closure
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
):
^
File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/groupcompress.py", line 1736, in _get_remaining_record_stream
yield from batcher.yield_factories(full_flush=True)
File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/groupcompress.py", line 1244, in yield_factories
block_read_memo, block = next(blocks)
~~~~^^^^^^^^
File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/groupcompress.py", line 1528, in _get_blocks
zdata = next(raw_records)
File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/pack_repo.py", line 2277, in get_raw_records
for _names, read_func in reader.iter_records():
~~~~~~~~~~~~~~~~~~~^^
File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/pack.py", line 320, in iter_records
self._read_format()
~~~~~~~~~~~~~~~~~^^
File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/pack.py", line 364, in _read_format
format = self._read_line()
File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/pack.py", line 288, in _read_line
line = self._source.readline()
File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/pack.py", line 252, in readline
self._next()
~~~~~~~~~~^^
File "/build/reproducible-path/breezy-3.3.11/breezy/bzr/pack.py", line 236, in _next
offset, data = next(self.readv_result)
~~~~^^^^^^^^^^^^^^^^^^^
File "/build/reproducible-path/breezy-3.3.11/breezy/transport/sftp.py", line 186, in request_and_yield_offsets
for (start, length), data in zip(requests, data_stream):
~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/paramiko/sftp_file.py", line 527, in readv
yield self.read(x[1])
~~~~~~~~~^^^^^^
File "/usr/lib/python3/dist-packages/paramiko/file.py", line 203, in read
new_data = self._read(read_size)
File "/usr/lib/python3/dist-packages/paramiko/sftp_file.py", line 182, in _read
data = self._read_prefetch(size)
File "/usr/lib/python3/dist-packages/paramiko/sftp_file.py", line 162, in _read_prefetch
self.sftp._read_response()
~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/usr/lib/python3/dist-packages/paramiko/sftp_client.py", line 887, in _read_response
t, data = self._read_packet()
~~~~~~~~~~~~~~~~~^^
File "/usr/lib/python3/dist-packages/paramiko/sftp.py", line 212, in _read_packet
x = self._read_all(4)
File "/usr/lib/python3/dist-packages/paramiko/sftp.py", line 196, in _read_all
x = self.sock.recv(n)
File "/build/reproducible-path/breezy-3.3.11/breezy/transport/ssh.py", line 193, in recv
return self.__socket.recv(n)
~~~~~~~~~~~~~~~~~~^^^
File "/build/reproducible-path/breezy-3.3.11/breezy/tests/fixtures.py", line 162, in signal_handler
raise TimeoutException(self.timeout_secs)
breezy.tests.fixtures.TimeoutException: 6000

----------------------------------------------------------------------
Ran 33064 tests in 6793.519s

FAILED (errors=1, known_failure_count=49)
2136 tests skipped
Missing feature 'Filesystem treats '\' as a directory separator.' skipped 5 tests.
Missing feature 'Missing tdb' skipped 6 tests.
Missing feature '_winreg' skipped 3 tests.
Missing feature 'apport.report' skipped 1 tests.
Missing feature 'attrib Windows command-line tool' skipped 1 tests.
Missing feature 'breezy._annotator_pyx' skipped 1 tests.
Missing feature 'breezy._chunks_to_lines_pyx' skipped 1 tests.
Missing feature 'breezy._readdir_pyx' skipped 2 tests.
Missing feature 'breezy.bzr._btree_serializer_pyx' skipped 23 tests.
Missing feature 'breezy.bzr._chk_map_pyx' skipped 1 tests.
Missing feature 'breezy.bzr._dirstate_helpers_pyx' skipped 104 tests.
Missing feature 'breezy.bzr._groupcompress_pyx' skipped 15 tests.
Missing feature 'breezy.bzr._knit_load_data_pyx' skipped 27 tests.
Missing feature 'breezy.bzr._rio_pyx' skipped 1 tests.
Missing feature 'case-insensitive case-preserving filesystem' skipped 20 tests.
Missing feature 'case-insensitive filesystem' skipped 1 tests.
Missing feature 'git executable' skipped 34 tests.
Missing feature 'gpg' skipped 3 tests.
Missing feature 'meliae.scanner' skipped 1 tests.
Missing feature 'pyinotify' skipped 16 tests.
Missing feature 'pywintypes' skipped 1 tests.
Missing feature 'quilt executable' skipped 21 tests.
Missing feature 'win32 platform' skipped 9 tests. >breezy.bzr.tests.test_smart_transport.ReadOnlyEndToEndTests.test_mkdir_error_readonly is leaking threads among 6 leaking tests.
1 non-main threads were left active in the end.
brz: warning: some compiled extensions could not be loaded; see ``brz help missing-extensions``
make[1]: *** [debian/rules:23: override_dh_auto_test] Error 1
make[1]: Leaving directory '/build/reproducible-path/breezy-3.3.11'
make: *** [debian/rules:15: binary-arch] Error 2
dpkg-buildpackage: error: debian/rules binary-arch subprocess returned exit status 2

...

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Jelmer,

* Jelmer Vernooij <jelmer@jelmer.uk> [250419 18:40]:

Is this repeatable or perhaps a flaky test?

https://buildd.debian.org/status/logs.php?pkg=breezy&ver=3.3.11-1&arch=s390x

It failed three out of three attempts on the buildd. So, not just
flaky I think?

Chris

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------0Dnz1HkoJMjKukFdFN5V7A72
Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64

SGkgSmVsbWVyLA0KDQpPbiBTYXQsIDE5IEFwciAyMDI1IDE2OjMxOjA2ICswMDAwIEplbG1l ciA9P3V0Zi04P1E/VmVybm9vPUM0PUIzPz0gDQo8amVsbWVyQGplbG1lci51az4gd3JvdGU6 DQo+IElzIHRoaXMgcmVwZWF0YWJsZSBvciBwZXJoYXBzIGEgZmxha3kgdGVzdD8NCg0KQW55 IHByb2dyZXNzIG9uIHRoaXMgYnVnPyBUaGUgRlRCRlMgaXMgaGluZGVyaW5nIHRoZSBtaWdy YXRpb24gdG8gDQp0ZXN0aW5nIG9mIHRoZSBmaXggZm9yIFJDIGJ1ZyAxMDk5MTIxLiBXZSdy ZSBydW5uaW5nIG91dCBvZiB0aW1lIGZvciB0cml4aWUuDQoNCk1pbmQgeW91LCBJIHZlcnkg bXVjaCBwcmVmZXIgYSB0YXJnZXRlZCBmb3IgYm90aCBpc3N1ZXMgKHJldmVydGluZyB0byAN CnRoZSB2ZXJzaW9uIGluIHRlc3RpbmcpIGJlY2F1c2UgdGhlIGRpZmYgYmV0d2VlbiB1bnN0 YWJsZSBhbmQgdGVzdGluZyBpcyANCmh1Z2UgKGFuZCB0aHVzIG1vc3QgbGlrZWx5IGluYXBw cm9wcmlhdGUgYXQgdGhpcyBzdGFnZSk6DQogIDQ3IGZpbGVzIGNoYW5nZWQsIDgzNSBpbnNl cnRpb25zKCspLCAzNTcyIGRlbGV0aW9ucygtKQ0KDQpQYXVsDQoNCg==

--------------0Dnz1HkoJMjKukFdFN5V7A72--

-----BEGIN PGP SIGNATURE-----

wsC7BAABCABvBYJoOwuJCRCcXJnrBb11CkcUAAAAAAAeACBzYWx0QG5vdGF0aW9u cy5zZXF1b2lhLXBncC5vcme6rbDQCJKfphKguzZsfka1jJme5Ij8gfy8/kRSaJ7n DBYhBFi2bUhza+k7BS3mcpxcmesFvXUKAADefggAoBrev3LdFUMMb+RGf5zSxCvZ WK3TkO7CTvK+091zmi++i2rJ8Zi6FlG2jSJz5prkp8RsMrf3FYkkUwADN7N5LvjO n6iOfKoV9Z0OdeWn2hlxiKuRUdwadR9M62kNyUJXsJP1kQa/4U/tmDE3D7KW29LP yzw01wFD+ou6RbGEe1QdOyiaf8zOnNhYgdn9obCXGLZjWlGGG0eohDyZfOp4vljV I3Uuji4avQUveH2R3l572nkfP/jQfgRtIZ5s/FYb/TK5TTPPnKi0LDNms3UaFQpw +Rd4mibwea+mrH3L3rligxvqUnZV2KjjCg8HbST9mpPdTyxL4reLyOKO3CXY9g==
=8hmU
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sat, May 31, 2025 at 04:00:41PM +0200, Paul Gevers wrote:

Hi Jelmer,

On Sat, 19 Apr 2025 16:31:06 +0000 Jelmer =?utf-8?Q?Vernoo=C4=B3?= ><jelmer@jelmer.uk> wrote:

Is this repeatable or perhaps a flaky test?

Any progress on this bug? The FTBFS is hindering the migration to
testing of the fix for RC bug 1099121. We're running out of time for
trixie.

Mind you, I very much prefer a targeted for both issues (reverting to
the version in testing) because the diff between unstable and testing
is huge (and thus most likely inappropriate at this stage):
47 files changed, 835 insertions(+), 3572 deletions(-)

Unfortunately not - I'm swamped by other stuff (upstream and otherwise), and skeptical this is a actually a new regression. I haven't had the time to actually go back and see whether this is reproducible with older versions though.

Most of the recent delta between unstable and testing was actually for addressing 1099121,
so I'm not sure how much you could reduce that (unless you were to prevent compilation
of some cython files rather than deleting them).

Jelmer

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)