Copy: ema@debian.org (Emanuele Rocca)

This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------13GlQZD8GTxiuqg00zL40IJj
Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64

SGVsbG8gU3RldmUsIGxpc3QsDQoNClNpbmNlIHRoZSBNaW5pZGViY29uZiBpbiBIYW1idXJn IGVhcmxpZXIgdGhpcyB5ZWFyLCB0aGUgc3VwcG9ydCBmb3IgDQphcm02NCBpbiB0aGUgRGVi aWFuIGxpdmUgaW1hZ2VzIGhhcyBzZWVuIGEgbG90IG9mIHByb2dyZXNzLg0KVG9nZXRoZXIg d2l0aCBFbWFudWVsZSBSb2NjYSwgSSB3YXMgYWJsZSB0byB0byBjb25maXJtIHRoYXQgbGl2 ZSBpbWFnZXMgDQp0aGF0IGFyZSBidWlsdCBuYXRpdmVseSBvbiBhcm02NCBjb250YWluIHRo ZSBzYW1lIGFzIHRoZSBjcm9zcy1idWlsdCANCmxpdmUgaW1hZ2VzIGZyb20gYW1kNjQuIEFk ZGl0aW9uYWxseSBzb21lIG9mIHRoZSBidWcgcmVwb3J0cyB0aGF0IHdlIA0KY3JlYXRlZCBo YXZlIGJlZW4gcmVzb2x2ZWQgc2luY2UgdGhlbi4NCg0KIEZyb20gbXkgcG9pbnQgb2Ygdmll dywgdGhlIHdlZWtseSBpbWFnZSBidWlsZGluZyAob3JnYW5pc2VkIGluIA0KbGl2ZS1zZXR1 cCkgY2FuIGJlIGV4dGVuZGVkIHRvIGluY2x1ZGUgYXJtNjQgd2FzIHdlbGwgWzFdLg0KDQpJ IGNvdWxkIHByZXBhcmUgYSBNUiwgYnV0IG5lZWQgYSBiaXQgbW9yZSBpbmZvOg0KSG93IGFy ZSB0aGUgbm9uLWFtZDY0IGltYWdlcyAoZS5nIHRoZSBjdXJyZW50IG5ldGluc3QtaW1hZ2Vz KSBjcmVhdGVkPyANCkFyZSB0aGV5IHRoZSByZXN1bHQgb2YgYSBjcm9zcy1idWlsZCwgb3Ig YXJlIHRoZXkgbmF0aXZlbHkgZ2VuZXJhdGVkIG9uIA0Kb3RoZXIgaG9zdHM/DQpUaGUgbGF0 ZXN0IGdub21lIGltYWdlIHRvb2sgMyBob3VycyBhY2NvcmRpbmcgdG8gdGhlIGxvZyBbMl0u DQpUaGUgY3Jvc3MtYnVpbGQgb24gbXkgY29tcHV0ZXIgdGFrZSA2IHRvIDEwIHRpbWVzIGFz IG11Y2ggY29tcHV0aW5nIA0KdGltZSwgc28gdGhhdCBtaWdodCBiZSBjb25zaWRlcmVkLg0K KEkndmUgYWxyZWFkeSBpZGVudGlmaWVkIG9uZSBjYXVzZSwgd2hpY2ggSSBpbnRlbmQgdG8g cmVtb3ZlIGxhdGVyKQ0KDQpJZiB0aGV5IGFyZSBjcm9zcy1idWlsZHMsIHBlcmhhcHMgaXQg d291bGQgYmUgd2lzZXIgdG8gc3RhcnQgd2l0aCBvbmUgDQppbWFnZSBhdCBmaXJzdCAoZS5n LiB0aGUgZ25vbWUgaW1hZ2UpIGFuZCB0aGVuIHNldCB1cCB0aGUgb3BlblFBIHRlc3Rpbmcg DQpjaGFpbiwgYmVmb3JlIHByb2dyZXNzaW5nIHRvIG90aGVyIHZhcmlhbnRzLg0KDQpXaXRo IGtpbmQgcmVnYXJkcywNClJvbGFuZCBDbG9idXMNCg0KWzFdIGh0dHBzOi8vZ2V0LmRlYmlh bi5vcmcvaW1hZ2VzL3dlZWtseS1saXZlLWJ1aWxkcy8NClsyXSANCmh0dHBzOi8vZ2V0LmRl Ymlhbi5vcmcvaW1hZ2VzL3dlZWtseS1saXZlLWJ1aWxkcy9hbWQ2NC9pc28taHlicmlkL2Rl Ymlhbi1saXZlLXRlc3RpbmctYW1kNjQtZ25vbWUubG9nIA0KKGZyb20gMDM6MTI6MzIgdG8g MDY6MDQ6NDgpDQoNClBTOiBEb24ndCBsZXQgdGhpcyBkZWxheSB0aGUgZm9jdXMgZnJvbSB0 aGUgdXBjb21pbmcgcG9pbnQgcmVsZWFzZXMgDQpwbGFubmVkIGZvciBuZXh0IHdlZWsuDQo=


--------------13GlQZD8GTxiuqg00zL40IJj--

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEUFVLM5Bdj7GSJEb+YsV8aqYUlb0FAmZ37ToACgkQYsV8aqYU lb0zrw/9EM7hlEb8eCsMBt86vzXEtJu6IxpSjjJTwa/OBU2bhrih6Hdy0L0XaMN9 R3F1im6L3shYOnsuNuOHMD0e7O3M2DrFP+vrTFBMJPl1wCD7cx8hKpwzmWq1bMGf vJDglO7GH0GHLyr5+It+tSNJm0WlcrA3N2+SX/g3fjchI8CrGbQn6qg2CvdRvBbY bLC5Uy4VfpXCclBSGE2qyIvCiGE9rudpR98vvnEv/06Z+pifx15EujEXBW0ScDdm 77sZHi8s4Fze3FwsyS+3sQ2LBnk1QR7+voMz3nQ/38TNERd/TrDve8sbpU1VwGqS uVUFBgClCr5pu9NvQYVQq3hLDujQCO5+lrMEEaVFlY1knENMxfsE19Fsc90j2kkB owJzcTO6OveVZjxDdTlGkQZpCeKOkRNhVJXOg/0vorI8BnlvcJe/1TLivtLKFUmk 7wp07gkDh86EQ+3vooMrs2lg6ncCm4ZDSV+LeWq+udFIV4e0OeSCXGTdMLUOITA8 Zh1nMnygtfPEsRx7qXtCIyyH35Oj6MzN9klnXfyIe5WplVyzH5iaQhHC7gSOAvAw w07yXQ7Lh9QqzUT6sIEYXRSgG1zGoNj1D0oq7NO7eTAImVjjJFRj5BfiqMKRfCf/ Ik/O4CFBbnSPa16X9O1fBiTNBaJbui7K6SxpHaABi4HhI180oWU=
=gX8O
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi,

Roland Clobus wrote:

The latest gnome image took 3 hours according to the log [2].
[2] https://get.debian.org/images/weekly-live-builds/amd64/iso-hybrid/debian-live-testing-amd64-gnome.log (from 03:12:32 to 06:04:48)

Interesting log. There are two time zones used in two kinds of time stamp.
The more frequently given stamps say "[2024-06-17 04:12:33]" to
"[2024-06-17 07:04:19]".

The first timestamp after ISO production is
[2024-06-17 04:58:02] lb binary_onie

It seems that most of the two hours after building the ISO are spent with activities which have nothing to do with the ISO production.
More than half of the overall 3 hours get spent with "lb chroot_archives"
runs. (I assume that those don't happen in debian-cd.)

In particular there is a strange time span of more than 1 hour from
[2024-06-17 04:58:32] lb chroot_archives source install
P: Configuring file /etc/apt/sources.list
to
[2024-06-17 06:03:35] lb source_live
P: Begin copying live-build configuration...
with not much reported inbetween.
Less than a minute before this time the ISO was produced.

Further i see a potential slow network connection at the very end.
[2024-06-17 06:32:48] lb chroot_archives source remove
P: Deconfiguring file /etc/apt/sources.list
Get:1 http://deb.debian.org/debian trixie InRelease [169 kB]
...
Get:13 http://deb.debian.org/debian trixie/non-free-firmware Icons (64x64) [29 B]
Fetched 40.9 MB in 31min 26s (21.7 kB/s)
Reading package lists...
[2024-06-17 07:04:19] lb chroot_hostname remove


Have a nice day :)

Thomas

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hey Roland!

On Sun, Jun 23, 2024 at 11:39:05AM +0200, Roland Clobus wrote:

Since the Minidebconf in Hamburg earlier this year, the support for arm64 in >the Debian live images has seen a lot of progress.
Together with Emanuele Rocca, I was able to to confirm that live images that >are built natively on arm64 contain the same as the cross-built live images >from amd64. Additionally some of the bug reports that we created have been >resolved since then.

Cool. :-)

From my point of view, the weekly image building (organised in live-setup) >can be extended to include arm64 was well [1].

That's awesome!

I could prepare a MR, but need a bit more info:
How are the non-amd64 images (e.g the current netinst-images) created? Are >they the result of a cross-build, or are they natively generated on other >hosts?

The installer images are all generated using debian-cd on casulana,
our big central amd64 with lots of fast storage. There's nothing that
needs native execution in debian-cd, so there's nothing that needs cross-building for installer images. Live images will be different, of
course.

The latest gnome image took 3 hours according to the log [2].
The cross-build on my computer take 6 to 10 times as much computing time, so >that might be considered.

Right. We build the live images in parallel inside a single VM on
casulana, so build times that you're seeing are not necessarily
representative. Last time I checked, a single gnome image build took
~1h. Running in paralllel lets us build multiple images in that 3h
period, so it's definitely worth it overall to do that.

For an example of the kind of performance cost of cross-building, we
*do* still build some amd64 images and cross-build some equivalent
arm64 openstack images on casulana. The last builds took:

* amd64 0h09m55s
* arm64 2h52m25s

which is approximately a factor of 17.

(I've already identified one cause, which I intend to remove later)

OK.

If they are cross-builds, perhaps it would be wiser to start with one image >at first (e.g. the gnome image) and then set up the openQA testing chain, >before progressing to other variants.

Sure, that would make sense. Building a full set of arm64 images
probably isn't feasible unless we do something to make the builds much
faster.

--
Steve McIntyre, Cambridge, UK. steve@einval.com Dance like no one's watching. Encrypt like everyone is.
- @torproject

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hey Roland,

On Wed, Jun 26, 2024 at 12:27:31PM +0200, Roland Clobus wrote:

On 23/06/2024 22:33, Steve McIntyre wrote:

Hey Roland!

On Sun, Jun 23, 2024 at 11:39:05AM +0200, Roland Clobus wrote:

Since the Minidebconf in Hamburg earlier this year, the support for arm64 in
the Debian live images has seen a lot of progress.

...

I could prepare a MR

The MR is at >https://salsa.debian.org/images-team/live-setup/-/merge_requests/5

Let's start only with a GNOME image at first.
I'll update the matching openQA tests, as soon as the new image appears.

ACK. As you've referenced, I'll also need to tweak some other bits in
the image setup. Maybe this weekend once we've done the point
releases...

--
Steve McIntyre, Cambridge, UK. steve@einval.com "C++ ate my sanity" -- Jon Rabone

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Jun 26, 2024 at 03:07:22PM +0100, Steve McIntyre wrote:

Hey Roland,

On Wed, Jun 26, 2024 at 12:27:31PM +0200, Roland Clobus wrote:

On 23/06/2024 22:33, Steve McIntyre wrote:

Hey Roland!

On Sun, Jun 23, 2024 at 11:39:05AM +0200, Roland Clobus wrote:

Since the Minidebconf in Hamburg earlier this year, the support for arm64 in
the Debian live images has seen a lot of progress.

...

I could prepare a MR

The MR is at >>https://salsa.debian.org/images-team/live-setup/-/merge_requests/5

Let's start only with a GNOME image at first.
I'll update the matching openQA tests, as soon as the new image appears.

ACK. As you've referenced, I'll also need to tweak some other bits in
the image setup. Maybe this weekend once we've done the point
releases...

I've merged everything and added that config and run a couple of times
to test. I'm seeing a (slow, as expected!) live image build OK, then
right at the end:

...
P: building the debian-installer
cp: cannot stat 'chroot/debian-installer/build/dest/cdrom/vmlinuz': No such file or directory
E: An unexpected failure occurred, exiting...
P: Begin unmounting filesystems...
P: Saving caches...
...

Did you test this to build an arm64 image yourself?

--
Steve McIntyre, Cambridge, UK. steve@einval.com Can't keep my eyes from the circling sky,
Tongue-tied & twisted, Just an earth-bound misfit, I...

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)