Hi,
在 2025/04/30 23:55, Salvatore Bonaccorso 写道:
Hi
We got a regression report in Debian after the update from 6.1.133 to
6.1.135. Melvin is reporting that discard/trimm trhough a RAID10 array
stalls idefintively. The full report is inlined below and originates
from https://bugs.debian.org/1104460 .
On Wed, Apr 30, 2025 at 04:46:50PM +0200, Melvin Vermeeren wrote:
Package: src:linux
Version: 6.1.135-1
Severity: important
Tags: upstream
X-Debbugs-Cc: vermeeren@vermwa.re
Dear Maintainer,
Upgrading from linux-image-6.1.0-33-powerpc64le (6.1.133-1) to
linux-image-6.1.0-34-powerpc64le (6.1.135-1) it appears there is a
serious regression bug related to discard/TRIM through a RAID10 array.
This only affects RAID10, RAID1 array on the same SSD device is not
affected. Array in question is a fairly standard RAID10 in 2far layout.
md127 : active raid10 dm-1[2] dm-0[0]
1872188416 blocks super 1.2 512K chunks 2 far-copies [2/2] [UU]
bitmap: 1/1 pages [64KB], 65536KB chunk
Any discard operation will result in quite a long kernel error. The
calling process will either segfault (swapon) or, more likely, be stuck
forever (Qemu, fstrim) in the D state per htop. The iostat utility
reports a %util of 100% for any device on top of (directly or
indirectly) of the RAID10 device, despite there being no read or write
requests to the devices or any other acitivty.
Stuck processes cannot be terminated or killed. Attempting to reboot
normally will result in a stuck machine on shutdown, so only a
REISUB-style reboot will work via procfs sysrq.
I have briefly diffed and inspected commits between the two kernel
versions and I suspect the commit below may be at fault. Do keep in mind >>> I have not verified this in any way, so I may be wrong.
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=4a05f7ae33716d996c5ce56478a36a3ede1d76f2
Thanks for the report, the commit relied on another commit
820455238366 ("md/raid10: switch to use md_account_bio() for io
accounting"), and it's wrong for v6.1. I'll send a revert soon.
Thanks,
Kuai
Considering this is shipped as part of a stable security update I
consider it quite a serious bug. Affected hosts will not boot up
cleanly, may not have swap, processes will freeze upon discard and clean >>> reboot it also not possible.
More logs available upon request.
Many thanks,
Melvin Vermeeren.
-- Package-specific info:
** Version:
Linux version 6.1.0-34-powerpc64le (debian-kernel@lists.debian.org)
(gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian)
2.40) #1 SMP Debian 6.1.135-1 (2025-04-25)
** Command line:
root=/dev/mapper/...-root ro quiet
** Not tainted
** Kernel log:
# /etc/fstab entry
/dev/.../swap none swap sw,discard=once 0 0
~# swapon -va
swapon: /dev/mapper/...-swap: found signature [pagesize=65536,
signature=swap]
swapon: /dev/mapper/...-swap: pagesize=65536, swapsize=17179869184,
devsize=17179869184
swapon /dev/mapper/...-swap
Segmentation fault
~# dmesg
...
[ 223.017257] kernel tried to execute user page (0) - exploit
attempt? (uid: 0)
[ 223.017287] BUG: Unable to handle kernel instruction fetch (NULL
pointer?)
[ 223.017301] Faulting instruction address: 0x00000000
[ 223.017326] Oops: Kernel access of bad area, sig: 11 [#1]
Hi,
在 2025/05/06 4:59, Antoine Beaupré 写道:
On 2025-05-05 22:36:07, Salvatore Bonaccorso wrote:
Hi Antoine,
On Mon, May 05, 2025 at 02:50:32PM -0400, Antoine Beaupré wrote:
On 2025-05-05 18:02:37, Salvatore Bonaccorso wrote:
On Mon, May 05, 2025 at 04:00:31PM +0200, Salvatore Bonaccorso wrote:
Hi Moritz,
On Mon, May 05, 2025 at 01:47:15PM +0200, Moritz Mühlenhoff wrote:
Am Wed, Apr 30, 2025 at 05:55:20PM +0200 schrieb Salvatore Bonaccorso:
Hi
We got a regression report in Debian after the update from 6.1.133 to
6.1.135. Melvin is reporting that discard/trimm trhough a RAID10 array
stalls idefintively. The full report is inlined below and originates
from https://bugs.debian.org/1104460 .
JFTR, we ran into the same problem with a few Wikimedia servers running
6.1.135 and RAID 10: The servers started to lock up once fstrim.service
got started. Full oops messages are available at https://phabricator.wikimedia.org/P75746
Thanks for this aditional datapoints. Assuming you wont be able to thest the other stable series where the commit d05af90d6218 ("md/raid10: fix missing discard IO accounting") went in, might you at
least be able to test the 6.1.y branch with the commit reverted again
and manually trigger the issue?
If needed I can provide a test Debian package of 6.1.135 (or 6.1.137)
with the patch reverted.
So one additional data point as several Debian users were reporting back beeing affected: One user did upgrade to 6.12.25 (where the commit was backported as well) and is not able to reproduce the issue there.
That would be me.
I can reproduce the issue as outlined by Moritz above fairly reliably in
6.1.135 (debian package 6.1.0-34-amd64). The reproducer is simple, on a RAID-10 host:
1. reboot
2. systemctl start fstrim.service
We're tracking the issue internally in:
https://gitlab.torproject.org/tpo/tpa/team/-/issues/42146
I've managed to workaround the issue by upgrading to the Debian package from testing/unstable (6.12.25), as Salvatore indicated above. There, fstrim doesn't cause any crash and completes successfully. In stable, it
just hangs there forever. The kernel doesn't completely panic and the machine is otherwise somewhat still functional: my existing SSH connection keeps working, for example, but new ones fail. And an `apt install` of another kernel hangs forever.
So likely at least in 6.1.y there are missing pre-requisites causing
the behaviour.
If you can test 6.1.135-1 with the commit 4a05f7ae33716d996c5ce56478a36a3ede1d76f2 reverted then you can fetch built packages at:
https://people.debian.org/~carnil/tmp/linux/1104460/
Can you also test with 4a05f7ae33716d996c5ce56478a36a3ede1d76f2 not
reverted, and also cherry-pick c567c86b90d4715081adfe5eb812141a5b6b4883?
Hi Yu,
Thanks for your followups.
On Tue, May 06, 2025 at 09:25:50AM +0800, Yu Kuai wrote:
Hi,
在 2025/05/06 4:59, Antoine Beaupré 写道:
On 2025-05-05 22:36:07, Salvatore Bonaccorso wrote:
Hi Antoine,
On Mon, May 05, 2025 at 02:50:32PM -0400, Antoine Beaupré wrote:
On 2025-05-05 18:02:37, Salvatore Bonaccorso wrote:
On Mon, May 05, 2025 at 04:00:31PM +0200, Salvatore Bonaccorso wrote:
Hi Moritz,
On Mon, May 05, 2025 at 01:47:15PM +0200, Moritz Mühlenhoff wrote:
Am Wed, Apr 30, 2025 at 05:55:20PM +0200 schrieb Salvatore Bonaccorso:
Hi
We got a regression report in Debian after the update from 6.1.133 to
6.1.135. Melvin is reporting that discard/trimm trhough a RAID10 array
stalls idefintively. The full report is inlined below and originates
from https://bugs.debian.org/1104460 .
JFTR, we ran into the same problem with a few Wikimedia servers running
6.1.135 and RAID 10: The servers started to lock up once fstrim.service
got started. Full oops messages are available at
https://phabricator.wikimedia.org/P75746
Thanks for this aditional datapoints. Assuming you wont be able to >> > > > > > thest the other stable series where the commit d05af90d6218
("md/raid10: fix missing discard IO accounting") went in, might you at
least be able to test the 6.1.y branch with the commit reverted again
and manually trigger the issue?
If needed I can provide a test Debian package of 6.1.135 (or 6.1.137)
with the patch reverted.
So one additional data point as several Debian users were reporting >> > > > > back beeing affected: One user did upgrade to 6.12.25 (where the
commit was backported as well) and is not able to reproduce the issue
there.
That would be me.
I can reproduce the issue as outlined by Moritz above fairly reliably in
6.1.135 (debian package 6.1.0-34-amd64). The reproducer is simple, on a
RAID-10 host:
1. reboot
2. systemctl start fstrim.service
We're tracking the issue internally in:
https://gitlab.torproject.org/tpo/tpa/team/-/issues/42146
I've managed to workaround the issue by upgrading to the Debian package
from testing/unstable (6.12.25), as Salvatore indicated above. There, >> > > > fstrim doesn't cause any crash and completes successfully. In stable, it
just hangs there forever. The kernel doesn't completely panic and the >> > > > machine is otherwise somewhat still functional: my existing SSH
connection keeps working, for example, but new ones fail. And an `apt >> > > > install` of another kernel hangs forever.
So likely at least in 6.1.y there are missing pre-requisites causing
the behaviour.
If you can test 6.1.135-1 with the commit
4a05f7ae33716d996c5ce56478a36a3ede1d76f2 reverted then you can fetch
built packages at:
https://people.debian.org/~carnil/tmp/linux/1104460/
Can you also test with 4a05f7ae33716d996c5ce56478a36a3ede1d76f2 not
reverted, and also cherry-pick c567c86b90d4715081adfe5eb812141a5b6b4883?
Thank you.
Antoine, Moritz,
https://people.debian.org/~carnil/tmp/linux/1104460-2/ contains a
build with 4a05f7ae33716d996c5ce56478a36a3ede1d76f2 *not* reverted and
with c567c86b90d4715081adfe5eb812141a5b6b4883 cherry-picked, can you
test this one as well?
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 546 |
| Nodes: | 16 (0 / 16) |
| Uptime: | 167:15:52 |
| Calls: | 10,385 |
| Calls today: | 2 |
| Files: | 14,057 |
| Messages: | 6,416,529 |