Hi,
I was also wondering about this Y2038 thingy and did some experiments.
I'm reporting what I found to document it, but don't see much actionable stuff here. Many thanks to Arnd Bergmann for his input.
Attempt #1: rebootstrap
Given that I develop rebootstrap, attempting to use it for a time64
bootstrap seemed quite natural. I've been talking to this with Steve
multiple times including DC22. The question was how to plug it in. In
the end I went for Arnd's suggestion to set DPKG_*_APPEND variables to
modify dpkg-buildflags. Not every package uses these flags, but a
majority do. For a survey, this is probably good enough.
Things went somewhat far. The first issue was zlib. There isn't much to
say about it: https://github.com/madler/zlib/issues/447
Then the libprelude build failed with symbol issues. It happens to build
a C++ library and its symbols are dependent on time types. Unsurprising.
This is a case where we do break ABI.
Then it occured to me that not failing does not imply not being affected right? You can break ABI without failing the build. So as a detection
method this has a significant risk of false negatives.
For just bootstrapping an incompatible time64 armhf, this method should mostly just work. Would someone like to have the resulting .debs? What
would do with them?
Attempt #2: reproducible
My other favourite QA tool is reproducible builds. So how about building
and comparing binary packages?
Unfortunately, build flags participate in build ids. That directly
implies a high risk of false positives and/or manual work.
Without further ado, here we go.
.debs unchanged:
* base-files
* db-defaults
* coreutils
* diffutils
* findutils
* libonig
* libidn
* libidn2
* libpipeline
* libpthread-stubs
* libsm
* gzip
Question: How many of these ignore dpkg-buildflags?
After this sample, it became clear that this method is also not very
helpful. While it shows that basically everything is affected, it
doesn't give a good handle on ABI breakage either.
Just for my understanding: what would be the correct way to handle this, under the assumption that the new symbol names that dpkg-shlibdeps
found are the ones we want? Is there a way to flag both the time32
and the tim64 set of symbols as correct, or would we have to have a per-architecture list of expected symbols for the new (still to be
decided) architecture name?
Then the libprelude build failed with symbol issues. It happens to build
a C++ library and its symbols are dependent on time types. Unsurprising.
This is a case where we do break ABI.
Then it occured to me that not failing does not imply not being affected right? You can break ABI without failing the build. So as a detection
method this has a significant risk of false negatives.
For just bootstrapping an incompatible time64 armhf, this method should mostly just work. Would someone like to have the resulting .debs? What
would do with them?
Attempt #2: reproducible
My other favourite QA tool is reproducible builds. So how about building
and comparing binary packages?
Unfortunately, build flags participate in build ids. That directly
implies a high risk of false positives and/or manual work.
Without further ado, here we go.
.debs unchanged:
* base-files
* db-defaults
* coreutils
This is good. One thing that I think has been missing from the discussion about armhf rebootstrap is the fact that we do have experience in Debian doing cross-cutting ABI transitions without having to change an
architecture name. We've done this at least three times in Debian history: the libc5->libc6 transition ('g' suffix - which still remains today in libpam0g!), the GCC 4.0 C++ ABI transition ('c2' suffix), and the 'long double' transition ('ldbl' suffix, and an example of doing this for an ABI transition that didn't affect all architectures).
The first of these didn't require analysis, we knew all shared libraries
were affected (and Debian was much smaller then).
The second required analysis but it was easy because the C++ ABI is very visible in ELF symbols.
The third is the best analogue to the time_t situation, because it involved changing the size of a type that could appear not just in ABIs of C++ libraries, but also C libraries.
And we did manage to survive that intact! https://lists.debian.org/debian-devel/2007/05/msg01173.html
Nowadays, I wonder if abi-compliance-checker would be usable for analyzing the ABIs of C libraries to accurately identify what needs to change for time64_t. I think it would be interesting to know from this how many shared libraries expose time_t size in their ABIs.
On Thu, Oct 20, 2022 at 09:42:58PM -0700, Steve Langasek wrote:
This is good. One thing that I think has been missing from the discussion about armhf rebootstrap is the fact that we do have experience in Debian doing cross-cutting ABI transitions without having to change an architecture name. We've done this at least three times in Debian history: the libc5->libc6 transition ('g' suffix - which still remains today in libpam0g!), the GCC 4.0 C++ ABI transition ('c2' suffix), and the 'long double' transition ('ldbl' suffix, and an example of doing this for an ABI transition that didn't affect all architectures).
Nowadays, I wonder if abi-compliance-checker would be usable for analyzing the ABIs of C libraries to accurately identify what needs to change for time64_t. I think it would be interesting to know from this how many shared
libraries expose time_t size in their ABIs.
Well, I didn't see anybody else expressing interest in this, so I had a go.
I've done a first-pass analysis of Ubuntu lunar/armhf using abi-compliance-checker. The results of my investigation, including scripts and detailed reports/logs, are here:
- abi-compliance-checker was used to analyze headers of 767 library
packages in the archive. Of those, 82 of 558 were successfully analyzed
and identified as library packages that would need patched for the
switch to 64-bit time_t to mark them as ABI-incompatible with the
previous version on armhf.
- an additional 209 packages could not be analyzed, because a-c-c failed
to compile the headers using its invocation of gcc. Assuming a roughly
equal distribution of ABI-changing vs non-ABI-changing libraries among
these that have failed to compile, we'd be looking at around 113 of 767
libraries that need changed.
- if we decide the level of effort to patch this many libraries is
worthwhile, to get an actual actionable list of libraries that need
patched for this transition would require getting the compile failures
down to zero. I did dig into the failures alphabetically from "a" to
"libe"; most of these failures were resolved by one of three methods.
- letting a-c-c know that this is a C library and to not try to compile
it with gcc's C++ frontend;
- excluding various headers that can't be included directly, either
because they are internal headers that are included via other headers
in the package, or because they have dependencies on other headers not
in the archive and can't be compiled (and therefore can't affect the
library ABI);
- pulling in additional undeclared package dependencies.
Even when applying these fixes, I still had 8 library packages whose
headers I wasn't able to get to compile. So there's still quite a bit
of work to do here.
- On this preliminary pass, I only included library packages that shipped
both headers and .so's, as this supposedly gives the best quality
results from a-c-c. To get a full list of ABI-breaking changes,
however, we need to include packages that ship only headers and not
.so's, to catch those libraries for which the .so is in a different
binary package (boost), and to catch plugin systems for executables
(apache).
While I'm currently working on this using Ubuntu, the results are largely applicable to Debian, and the script itself certainly is. Note however that there are a couple of bugs in abi-compliance-checker that I've patched in Ubuntu, bug #951076 and bug #1030540, which you would want to have fixed before trying to run this analysis in Debian. Cc:ing the a-c-c maintainer for awareness of this.
Does doing an ABI transition of ~113 libraries seem tractable to folks?
If
there's interest in moving forward on this, I'd say the next step should be to take it to debian-devel for broader discussion/signoff. I would also
move my scripts into a git repository that folks can contribute to -- adding the necessary overrides for a-c-c for each library, so we can get an authoritative list of ABI breakages, is very parallelizable.
On 2023-02-04 21:42 -0800, Steve Langasek wrote:
On Thu, Oct 20, 2022 at 09:42:58PM -0700, Steve Langasek wrote:
Does doing an ABI transition of ~113 libraries seem tractable to folks?
It certainly provides evidence for the idea that this is not
completely intractable, which I think many people (including me)
worried was the case initially.
On Wed, Feb 15, 2023, at 18:08, Wookey wrote:
On 2023-02-04 21:42 -0800, Steve Langasek wrote:
On Thu, Oct 20, 2022 at 09:42:58PM -0700, Steve Langasek wrote:
Does doing an ABI transition of ~113 libraries seem tractable to folks?
It certainly provides evidence for the idea that this is not
completely intractable, which I think many people (including me)
worried was the case initially.
When I did a similar analysis a few years ago using just pattern-matching
on header files, the result was that more than half the total packages
in Debian depended on at least one other package that needed an ABI transition, which in my mind made it unrealistic.
On Monday, 20 February 2023 16:24:37 CET Arnd Bergmann wrote:
On Wed, Feb 15, 2023, at 18:08, Wookey wrote:
On 2023-02-04 21:42 -0800, Steve Langasek wrote:
On Thu, Oct 20, 2022 at 09:42:58PM -0700, Steve Langasek wrote:
Does doing an ABI transition of ~113 libraries seem tractable to folks?
It certainly provides evidence for the idea that this is not
completely intractable, which I think many people (including me)
worried was the case initially.
When I did a similar analysis a few years ago using just pattern-matching on header files, the result was that more than half the total packages
in Debian depended on at least one other package that needed an ABI transition, which in my mind made it unrealistic.
If you do it in the early stage of Trixie's dev cycle, would it still be unrealistic? It may be a bumpy ride and take a while, but I don't see an immediate issue with that. Sid may finally become Unstable again ;-P
But *when* you do it, is quite relevant. If you/we are only a few months away
from the Trixie Freeze, then it's probably not a good idea.
But if we're 1-1.5 years before that, there's plenty of time to fix things.
Or is that too simplistic on my part?
Is there a place one can grep all buildlogs?
On 2023-02-04 21:42 -0800, Steve Langasek wrote:
Yes I think we should proceed with this analysis on debian to get a
better handle on just how many libraries we are looking at.
On 2023-02-15 17:08 +0000, Wookey wrote:
On 2023-02-04 21:42 -0800, Steve Langasek wrote:
Yes I think we should proceed with this analysis on debian to get a
better handle on just how many libraries we are looking at.
OK. We have over 10,000 *-dev package in debian so this took a while,
but I've now finished an initial pass.
The results are:
Total -dev packages: 10323
Time_t changes ABI: 329
LFS changes ABI: 58 0.6%
ABI unchanged: 1840 5.6%
Compilation failed: 1637
No headers in pkg: 3852
No library in pkg: 5086
total analysed: 10249
(note that the no-headers and no-libs categories are not exclusive
- the other classifications are)
Not quite sure where we lost 74 packages down the back of the sofa but at 0.7% it's noise.
So we have about 5000 packages which can basically be ignored for this purpose:
golang* (1943)
librust* (1955) - source-only, no dynamic linking, no .so's in any package and libghc* (1065) - changes ABI at drop of hat (every new version) anyway.
Of the remaining 5360, 5237 have .so files to check. Of those 329
change ABI and another 58 are not built with LFS currently
so also change ABI due to that. That's 387 (7%).
34% are fine. And an annoyingly large 1637 (30%) did not build under Abi-compliance-checker to determine one way or the other. Assuming 7%
of those are a problem that's another 114 packages.
I've yet to determine how many of the 'no-libs/no-headers' packages
actually expose an ABI we need to worry about. There will be some
more, from stuff like boost and Qt.
So the scale of the transition task in Debian is quite a lot bigger than in Ubuntu: Probably 400-500 packages.
I've linked my lists on the wiki page. https://wiki.debian.org/ReleaseGoals/64bit-time
I believe Steve L (who has done most of the heavy lifting on the
script) is running a check too and will have some results soon.
Wookey
--
Principal hats: Debian, Wookware, ARM
http://wookware.org/
On 2023-02-04 21:42 -0800, Steve Langasek wrote:
On Thu, Oct 20, 2022 at 09:42:58PM -0700, Steve Langasek wrote:
This is good. One thing that I think has been missing from the discussion
about armhf rebootstrap is the fact that we do have experience in Debian doing cross-cutting ABI transitions without having to change an architecture name. We've done this at least three times in Debian history:
the libc5->libc6 transition ('g' suffix - which still remains today in libpam0g!), the GCC 4.0 C++ ABI transition ('c2' suffix), and the 'long double' transition ('ldbl' suffix, and an example of doing this for an ABI
transition that didn't affect all architectures).
I feel like I should already know this, but after a bit of thought,
and reading the transitions wiki page: https://wiki.debian.org/Teams/ReleaseTeam/Transitions I do not
understand how the suffixes are used in these transitions.
That wiki page does not appear to mention adding suffixes to package
names. It appears to only document the process where the version is
just bumped, everything involved is rebuilt in experimental and then re-uploaded to unstable once we are ready to make the change.
I presume that the suffix is used for transitions involving a large
number of packages, in order to keep both the old-ABI and new-ABI
version of the packages around in parallel (and conflicting with each
other?) for a while which is necessary for larger transitions because
doing it all in one go would take too long, or get entangled with
uploads from the unware that might cause breakage, or something?
Just rebuilding and not renaming sounds like a much smoother process,
because as soon as we rename then there is a load of renaming in
reverse dependencies too, and there is presumably a set of rules about
when conflicts are added and other details that need to be got right.
Sorry for the long radio silence on this; I had identified some issues with my prior report and wanted to rerun it with some fixes, and that analysis took rather longer than expected (mainly due to infrastructure slowness).
Refreshed output for Ubuntu is here:
https://people.canonical.com/~vorlon/armhf-time_t/
So comparing with Wookey's results, I see:
Total -dev packages tested: 4603
Time_t changes ABI: 349
LFS changes ABI: 71
ABI unchanged: 2085
Compilation failed: 2098
So certainly this appears to be the same order of magnitude as Wookey's separate analysis.
The other question is what impact this has on the archive as a whole, with triggering of rebuild transitions. Looking at the
reverse-build-dependencies of those 349 packages gives:
$ while read pkg; do grep-dctrl -n -sPackage -w -FBuild-Depends $pkg /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lunar_*Sources ; done < abi-breaks | sort -u | wc -l
3626
$
NB this is a bit of an undercount as a result of packages that ship headers but aren't the thing being directly build-depended on. A more accurate
count is probably to look at binary packages, from the same source as the -dev package, that the -dev package depends on and look at reverse-dependencies of those. But for a first pass, here's where we are.
There are also roughly 2100 packages for which compilation failed. To know which packages actually have ABI changes and require transitions, we would need to work through this list and get the count down to 0.
But as a first-order approximation, this looks doable to me as a transition we could manage.
What do other folks think? Are you on board with proposing this to debian-devel? Anyone want to help add the necessary quirks to drive down
the list of compilation failures?
On Tue, Mar 21, 2023 at 03:00:41AM +0000, Wookey wrote:
On 2023-02-15 17:08 +0000, Wookey wrote:
On 2023-02-04 21:42 -0800, Steve Langasek wrote:
Yes I think we should proceed with this analysis on debian to get a better handle on just how many libraries we are looking at.
OK. We have over 10,000 *-dev package in debian so this took a while,
but I've now finished an initial pass.
The results are:
Total -dev packages: 10323
Time_t changes ABI: 329
LFS changes ABI: 58 0.6%
ABI unchanged: 1840 5.6%
Compilation failed: 1637
No headers in pkg: 3852
No library in pkg: 5086
total analysed: 10249
(note that the no-headers and no-libs categories are not exclusive
- the other classifications are)
Not quite sure where we lost 74 packages down the back of the sofa but at 0.7% it's noise.
So we have about 5000 packages which can basically be ignored for this purpose:
golang* (1943)
librust* (1955) - source-only, no dynamic linking, no .so's in any package and libghc* (1065) - changes ABI at drop of hat (every new version) anyway.
Of the remaining 5360, 5237 have .so files to check. Of those 329
change ABI and another 58 are not built with LFS currently
so also change ABI due to that. That's 387 (7%).
34% are fine. And an annoyingly large 1637 (30%) did not build under Abi-compliance-checker to determine one way or the other. Assuming 7%
of those are a problem that's another 114 packages.
I've yet to determine how many of the 'no-libs/no-headers' packages actually expose an ABI we need to worry about. There will be some
more, from stuff like boost and Qt.
So the scale of the transition task in Debian is quite a lot bigger than in Ubuntu: Probably 400-500 packages.
I've linked my lists on the wiki page. https://wiki.debian.org/ReleaseGoals/64bit-time
I believe Steve L (who has done most of the heavy lifting on the
script) is running a check too and will have some results soon.
Wookey
--
Principal hats: Debian, Wookware, ARM
http://wookware.org/
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 489 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 14:27:28 |
| Calls: | 9,665 |
| Files: | 13,712 |
| Messages: | 6,167,663 |