On Sat, Oct 12, 2024 at 10:12:56AM +0200, Michał Górny wrote:
As a side notice, the existing versions would probably remain as-is
until removal, since there's really no gain in splitting them, given
we'd have to retain compatibility with existing depstrings.
so no dev-lang/python3_13t? nothing crucial but would have been nice
Historically, all versions of CPython were slotted in a single package,
i.e.:
dev-lang/python:3.N
This approach has been causing a major annoyance for users -- due to
Portage "greedy" upgrade behavior, any time a new Python version was keyworded, Portage insisted on installing it, even though user's
selected targets did not request the specific version. The potentially
worst consequence of that would be random user scripts stopping to work,
as they suddenly start using new Python, while all their dependencies
are still installed per PYTHON_TARGETS.
Upstream has recently added freethreading support to CPython. Since
this support is not ABI compatible with the regular build, we need to introduce a separate target for it, and to package it separately.
In the planned patchset, I've already put it as a separate package (dev- lang/python-freethreading), because otherwise Portage would insist
on upgrading to it!
However, I think the cleanest way forward would be to stop slotting
CPython like this, and instead have a separate package for each version,
just like the vast majority of distributions do, i.e.:
dev-lang/python3_N
This naturally means that only the specific version requested (e.g. via targets) would be installed, and no cross-slot autoupgrades would
happen. Ideally, I'd like to start doing that with Python 3.14 whose
first alpha is expected next week. Depending on how they handle freethreading, we'd end up having the first or both of:
dev-lang/python3_14
dev-lang/python3_14t
(Alternatives: python-3_14, python-freethreading-3_14? Though I think following PYTHON_TARGETS is cleaner here.)
As a side notice, the existing versions would probably remain as-is
until removal, since there's really no gain in splitting them, given
we'd have to retain compatibility with existing depstrings.
Comments?
On Sat, 12 Oct 2024, Ulrich Mueller wrote:
IMHO this would abuse the package name for information that absolutely doesn't belong there. It belongs in PV or SLOT.
To me it seems that you try to work around a problem (greedy upgrade behaviour) that should really be solved in the package manager.
On Sat, 12 Oct 2024, Ulrich Mueller wrote:
IMHO this would abuse the package name for information that absolutely doesn't belong there. It belongs in PV or SLOT.
To me it seems that you try to work around a problem (greedy upgrade behaviour) that should really be solved in the package manager.
How about introducing something like PROPERTIES="separate-slot" instead, which would instruct the PM to treat the slot like a separate package?
On Sat, 12 Oct 2024, Michał Górny wrote:
IMHO this would abuse the package name for information that absolutely doesn't belong there. It belongs in PV or SLOT.
To me it seems that you try to work around a problem (greedy upgrade behaviour) that should really be solved in the package manager.
In my opinion, it's the other way around. We have slots, that are a fit solution for packages that are roughly compatible between every major release, and we keep abusing them for every single thing we can bend
enough to make it fit.
So are you saying that Python versions aren't "roughly compatible"
between releases? Like, they're a completely different language?
On Sat, 12 Oct 2024, Michał Górny wrote:
IMHO this would abuse the package name for information that absolutely
doesn't belong there. It belongs in PV or SLOT.
To me it seems that you try to work around a problem (greedy upgrade
behaviour) that should really be solved in the package manager.
In my opinion, it's the other way around. We have slots, that are a fit solution for packages that are roughly compatible between every major release, and we keep abusing them for every single thing we can bend
enough to make it fit.
On Sat, Oct 12, 2024, 06:23 Michał Górny <mgorny@gentoo.org> wrote:
On Sat, 2024-10-12 at 12:13 +0200, Ulrich Mueller wrote:
On Sat, 12 Oct 2024, Michał Górny wrote:
absolutelyIMHO this would abuse the package name for information that
doesn't belong there. It belongs in PV or SLOT.
To me it seems that you try to work around a problem (greedy upgrade behaviour) that should really be solved in the package manager.
fitIn my opinion, it's the other way around. We have slots, that are a
solution for packages that are roughly compatible between every major release, and we keep abusing them for every single thing we can bend enough to make it fit.
So are you saying that Python versions aren't "roughly compatible" between releases? Like, they're a completely different language?
They're not compatible in the sense "if I install package for
python3.12, it will work in python3.13", because every version uses
a separate package tree. Upstream has effectively made every version separate. On top of that, the same "minor" version can have a PyPy variation and a "freethreading" variation now.
--
Best regards,
Michał Górny
How will this affect ebuilds that can use any version of Python 3? Will it need a giant or-block in DEPEND?
On Sat, 2024-10-12 at 12:13 +0200, Ulrich Mueller wrote:
On Sat, 12 Oct 2024, Michał Górny wrote:
absolutelyIMHO this would abuse the package name for information that
doesn't belong there. It belongs in PV or SLOT.
To me it seems that you try to work around a problem (greedy upgrade behaviour) that should really be solved in the package manager.
fitIn my opinion, it's the other way around. We have slots, that are a
solution for packages that are roughly compatible between every major release, and we keep abusing them for every single thing we can bend enough to make it fit.
So are you saying that Python versions aren't "roughly compatible"
between releases? Like, they're a completely different language?
They're not compatible in the sense "if I install package for
python3.12, it will work in python3.13", because every version uses
a separate package tree. Upstream has effectively made every version separate. On top of that, the same "minor" version can have a PyPy
variation and a "freethreading" variation now.
--
Best regards,
Michał Górny
Hello,
Historically, all versions of CPython were slotted in a single
package, i.e.:
dev-lang/python:3.N
This approach has been causing a major annoyance for users -- due to
Portage "greedy" upgrade behavior, any time a new Python version was keyworded, Portage insisted on installing it, even though user's
selected targets did not request the specific version. The
potentially worst consequence of that would be random user scripts
stopping to work, as they suddenly start using new Python, while all
their dependencies are still installed per PYTHON_TARGETS.
Upstream has recently added freethreading support to CPython. Since
this support is not ABI compatible with the regular build, we need to introduce a separate target for it, and to package it separately.
In the planned patchset, I've already put it as a separate package
(dev- lang/python-freethreading), because otherwise Portage would
insist on upgrading to it!
However, I think the cleanest way forward would be to stop slotting
CPython like this, and instead have a separate package for each
version, just like the vast majority of distributions do, i.e.:
dev-lang/python3_N
This naturally means that only the specific version requested (e.g.
via targets) would be installed, and no cross-slot autoupgrades would
happen. Ideally, I'd like to start doing that with Python 3.14 whose
first alpha is expected next week. Depending on how they handle freethreading, we'd end up having the first or both of:
dev-lang/python3_14
dev-lang/python3_14t
(Alternatives: python-3_14, python-freethreading-3_14? Though I think following PYTHON_TARGETS is cleaner here.)
As a side notice, the existing versions would probably remain as-is
until removal, since there's really no gain in splitting them, given
we'd have to retain compatibility with existing depstrings.
Comments?
Its already a serious chore keeping the python ebuilds in syncLife choices.
with ::gentoo and if you start to double the versions that will only be
more ridiculous.
On Sat, 2024-10-12 at 11:59 +0200, Ulrich Mueller wrote:
On Sat, 12 Oct 2024, Michał Górny wrote:
version,However, I think the cleanest way forward would be to stop slotting CPython like this, and instead have a separate package for each
just like the vast majority of distributions do, i.e.:
dev-lang/python3_N
That other distributions do it that way is not an argument because most other distributions don't have slots.
This naturally means that only the specific version requested (e.g. via targets) would be installed, and no cross-slot autoupgrades would
happen. Ideally, I'd like to start doing that with Python 3.14 whose first alpha is expected next week. Depending on how they handle freethreading, we'd end up having the first or both of:
dev-lang/python3_14
dev-lang/python3_14t
IMHO this would abuse the package name for information that absolutely doesn't belong there. It belongs in PV or SLOT.
To me it seems that you try to work around a problem (greedy upgrade behaviour) that should really be solved in the package manager.
In my opinion, it's the other way around. We have slots, that are a fit solution for packages that are roughly compatible between every major release, and we keep abusing them for every single thing we can bend
enough to make it fit.
Greedy upgrade behavior makes sense when you have packages where :=
and :* operators are applicable. It doesn't make sense when every
single dependency is expected to specify an explicit slot. In fact,
when you are never supposed to depend on the package without specifying
a slot, why would you think slots are the right solution?
In fact, the "freethreading" variation already implies that there could
be more than one "slot" per package version.
It's as meaningless as having sqlite3 packaged as sqlite:3, or gtk that
is now randomly split between gtk+:2, gtk+:3 and gtk:4.
--
Best regards,
Michał Górny
</div><div dir="auto">I didn't like this idea at first, but I'm warming up to it.<br><br><div class="gmail_quote" dir="auto"><div dir="ltr" class="gmail_attr">On Sat, Oct 12, 2024, 06:05 Michał Górny <<a href="mailto:mgorny@gentoo.org">mgorny@gentoo.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Sat, 2024-10-12 at 11:59 +0200, Ulrich Mueller wrote:<br>
How will the upgrade path look, from e.g. 3.16 to 3.17? What will the typical Gentoo user experience be? What will the
experience be for a Python developer with a local overlay containing custom Python packages?
I didn't like this idea at first, but I'm warming up to it.
On Sat, Oct 12, 2024, 06:05 Michał Górny <mgorny@gentoo.org> wrote:
On Sat, 2024-10-12 at 11:59 +0200, Ulrich Mueller wrote:
On Sat, 12 Oct 2024, Michał Górny wrote:
However, I think the cleanest way forward would be to stop slotting CPython like this, and instead have a separate package for each version, just like the vast majority of distributions do, i.e.:
dev-lang/python3_N
That other distributions do it that way is not an argument because most other distributions don't have slots.
This naturally means that only the specific version requested (e.g. via targets) would be installed, and no cross-slot autoupgrades would happen. Ideally, I'd like to start doing that with Python 3.14 whose first alpha is expected next week. Depending on how they handle freethreading, we'd end up having the first or both of:
dev-lang/python3_14
dev-lang/python3_14t
IMHO this would abuse the package name for information that absolutely doesn't belong there. It belongs in PV or SLOT.
To me it seems that you try to work around a problem (greedy upgrade behaviour) that should really be solved in the package manager.
In my opinion, it's the other way around. We have slots, that are a fit
solution for packages that are roughly compatible between every major
release, and we keep abusing them for every single thing we can bend
enough to make it fit.
Greedy upgrade behavior makes sense when you have packages where :=
and :* operators are applicable. It doesn't make sense when every
single dependency is expected to specify an explicit slot. In fact,
when you are never supposed to depend on the package without specifying
a slot, why would you think slots are the right solution?
In fact, the "freethreading" variation already implies that there could
be more than one "slot" per package version.
It's as meaningless as having sqlite3 packaged as sqlite:3, or gtk that
is now randomly split between gtk+:2, gtk+:3 and gtk:4.
--
Best regards,
Michał Górny
IMHO this would abuse the package name for information that absolutely doesn't belong there. It belongs in PV or SLOT.
To me it seems that you try to work around a problem (greedy upgrade behaviour) that should really be solved in the package manager.
In my opinion, it's the other way around. We have slots, that are a fit solution for packages that are roughly compatible between every major release, and we keep abusing them for every single thing we can bend
enough to make it fit.
It's as meaningless as having sqlite3 packaged as sqlite:3, or gtk that
is now randomly split between gtk+:2, gtk+:3 and gtk:4.
In fact, when you are never supposed to depend on the package without specifying a slot, why would you think slots are the right solution?
This approach has been causing a major annoyance for users -- due to
Portage "greedy" upgrade behavior, any time a new Python version was keyworded, Portage insisted on installing it, even though user's
selected targets did not request the specific version. The potentially
worst consequence of that would be random user scripts stopping to work,
as they suddenly start using new Python, while all their dependencies
are still installed per PYTHON_TARGETS.
This naturally means that only the specific version requested (e.g. via targets) would be installed, and no cross-slot autoupgrades would
happen. Ideally, I'd like to start doing that with Python 3.14 whose
first alpha is expected next week. Depending on how they handle freethreading, we'd end up having the first or both of:
dev-lang/python3_14
dev-lang/python3_14t
(Alternatives: python-3_14, python-freethreading-3_14? Though I think following PYTHON_TARGETS is cleaner here.)
As a side notice, the existing versions would probably remain as-is
until removal, since there's really no gain in splitting them, given
we'd have to retain compatibility with existing depstrings.
Comments?
On Sat, 2024-10-12 at 10:12 +0200, Michał Górny wrote:
This naturally means that only the specific version requested (e.g. via targets) would be installed, and no cross-slot autoupgrades would
happen. Ideally, I'd like to start doing that with Python 3.14 whose
first alpha is expected next week. Depending on how they handle freethreading, we'd end up having the first or both of:
dev-lang/python3_14
dev-lang/python3_14t
(Alternatives: python-3_14, python-freethreading-3_14? Though I think following PYTHON_TARGETS is cleaner here.)
As a side notice, the existing versions would probably remain as-is
until removal, since there's really no gain in splitting them, given
we'd have to retain compatibility with existing depstrings.
Comments?
Given all the opposition, I retract this.
freethreading or Python 3.14 on their systems, it's on you.
--
Best regards,
Michał Górny
Hello,
Historically, all versions of CPython were slotted in a single package,
i.e.:
dev-lang/python:3.N
This approach has been causing a major annoyance for users -- due to
Portage "greedy" upgrade behavior, any time a new Python version was keyworded, Portage insisted on installing it, even though user's
selected targets did not request the specific version. The potentially
worst consequence of that would be random user scripts stopping to work,
as they suddenly start using new Python, while all their dependencies
are still installed per PYTHON_TARGETS.
Upstream has recently added freethreading support to CPython. Since
this support is not ABI compatible with the regular build, we need to introduce a separate target for it, and to package it separately.
In the planned patchset, I've already put it as a separate package (dev- lang/python-freethreading), because otherwise Portage would insist
on upgrading to it!
However, I think the cleanest way forward would be to stop slotting
CPython like this, and instead have a separate package for each version,
just like the vast majority of distributions do, i.e.:
dev-lang/python3_N
This naturally means that only the specific version requested (e.g. via targets) would be installed, and no cross-slot autoupgrades would
happen. Ideally, I'd like to start doing that with Python 3.14 whose
first alpha is expected next week. Depending on how they handle freethreading, we'd end up having the first or both of:
dev-lang/python3_14
dev-lang/python3_14t
(Alternatives: python-3_14, python-freethreading-3_14? Though I think following PYTHON_TARGETS is cleaner here.)
As a side notice, the existing versions would probably remain as-is
until removal, since there's really no gain in splitting them, given
we'd have to retain compatibility with existing depstrings.
Comments?
However, I think the cleanest way forward would be to stop slotting
CPython like this, and instead have a separate package for each version, just like the vast majority of distributions do, i.e.:
dev-lang/python3_N
As others have noted, such a proposal needs specific arguments as to why SLOTs aren't a good fit. I agree with you that they're not always a good
fit -- SQLite and libxml2 are good examples you gave downthread, but
the onus is on the one making the proposal.
Now, for Python, there's a few disadvantages:
* losing the ordering on PV for e.g. has_version (we could add a helper
in python-utils-r1 for this);
* losing the ability to consistently set package.use/package.env for all Pythons, like always enabling PGO or tests;
* disruption to scripts which have reasonably assumed we'd always have a dev-lang/python (we'd need to do something like we have planned for
pkgmoves, I think -- make Portage know about it and suggest alternatives intelligently/rewrite it transparently when given as an argument).
This naturally means that only the specific version requested (e.g. via targets) would be installed, and no cross-slot autoupgrades would
happen. Ideally, I'd like to start doing that with Python 3.14 whose
first alpha is expected next week. Depending on how they handle freethreading, we'd end up having the first or both of:
dev-lang/python3_14
dev-lang/python3_14t
It's worth noting that we *do* this for pypy, but we retain
dev-python/pypy3. I'm not a huge fan of it there but I know why we have
it -- so that one can test new versions of pypy in parallel even when
they supply the same implementation/version of the Python language.
On Mon, 2024-10-14 at 01:43 +0100, Sam James wrote:
However, I think the cleanest way forward would be to stop slotting
CPython like this, and instead have a separate package for each version, >> > just like the vast majority of distributions do, i.e.:
dev-lang/python3_N
As others have noted, such a proposal needs specific arguments as to why
SLOTs aren't a good fit. I agree with you that they're not always a good
fit -- SQLite and libxml2 are good examples you gave downthread, but
the onus is on the one making the proposal.
Now, for Python, there's a few disadvantages:
* losing the ordering on PV for e.g. has_version (we could add a helper
in python-utils-r1 for this);
I don't get this. You can't use has_version directly without specifying
the slot, because it's going to match different versions. And there's
no real difference between specifying a slot and a different package
name.
Well, unless you mean doing a meaningless has_version match for the sake
of it. Then, yes, unslotting fixes that -- i.e. removes that ability.
* losing the ability to consistently set package.use/package.env for all
Pythons, like always enabling PGO or tests;
We aren't losing it, you just need to repeat it. Just like right now
you can apply these per-slot or restrict version ranges, so there's no guarantee of consistency either.
* disruption to scripts which have reasonably assumed we'd always have a
dev-lang/python (we'd need to do something like we have planned for
pkgmoves, I think -- make Portage know about it and suggest alternatives
intelligently/rewrite it transparently when given as an argument).
Yes, this is a fair point, and the logic in pkgcheck is pretty horibble,
so I guess going for slotting just to avoid having to fix that
and deploy the fix makes sense.
This naturally means that only the specific version requested (e.g. via
targets) would be installed, and no cross-slot autoupgrades would
happen. Ideally, I'd like to start doing that with Python 3.14 whose
first alpha is expected next week. Depending on how they handle
freethreading, we'd end up having the first or both of:
dev-lang/python3_14
dev-lang/python3_14t
It's worth noting that we *do* this for pypy, but we retain
dev-python/pypy3. I'm not a huge fan of it there but I know why we have
it -- so that one can test new versions of pypy in parallel even when
they supply the same implementation/version of the Python language.
Technically, we could merge PyPy into a single package, as long as we
use verisons such as 2.7.7.3.17:2.7, 3.10.7.3.17:3.10, etc.
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 546 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 51:36:42 |
| Calls: | 10,397 |
| Calls today: | 5 |
| Files: | 14,067 |
| Messages: | 6,417,338 |
| Posted today: | 1 |