On Fri, Oct 21, 2016 at 08:16:31PM +0200, John Paul Adrian Glaubitz wrote:

Hi!

We have recently run into a rather odd regression with glibc on Debian/sh4. Starting with glibc 2.24, file offsets are apparently wrong which results into all weird side effects.

One of these effects is that many applications error out with "File too large",
for example firebird3.0 during build:

sh -x -c "lockfile -1 /<<PKGBUILDDIR>>/gen/Release/firebird/bin/build-db.lock && /<<PKGBUILDDIR>>/gen/Release/firebird/bin/isql -q -i
/<<PKGBUILDDIR>>/src/dbs/metadata.sql; res=\$?; rm -f /<<PKGBUILDDIR>>/gen/Release/firebird/bin/build-db.lock; exit \$res"
+ lockfile -1 /<<PKGBUILDDIR>>/gen/Release/firebird/bin/build-db.lock
ERROR: ld.so: object 'libeatmydata.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
+ /<<PKGBUILDDIR>>/gen/Release/firebird/bin/isql -q -i /<<PKGBUILDDIR>>/src/dbs/metadata.sql
can't format message 17:0 -- message file /usr/lib/sh4-linux-gnu/firebird/3.0/firebird.msg not found
I/O error during "write" operation for file "/<<PKGBUILDDIR>>/gen/metadata.fdb"
-Error while trying to write to file
-File too large
can't format message 17:120 -- message file /usr/lib/sh4-linux-gnu/firebird/3.0/firebird.msg not found
+ res=1
+ rm -f /<<PKGBUILDDIR>>/gen/Release/firebird/bin/build-db.lock
+ exit 1

Or the systemd-journal:

[ 13.088000] systemd-journald[524]: Failed to create new system journal: File too large

Another side effect is that mkfs does not work properly anymore since it writes the superblock
at an incorrect offset. With a properly working glibc, the superblock is written at an offset
of 0x400 while with the broken glibc (2.24), the superblock starts at 0x000. As a result,
it's not possible to create a usable filesystem on affected machines:

root@tirpitz:~> mkfs.ext4 /dev/sda2
mke2fs 1.43.3 (04-Sep-2016)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Creating filesystem with 16252928 4k blocks and 4063232 inodes
Filesystem UUID: 7d9bf1ee-a913-45d0-895b-4032020c1f3b
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424

Allocating group tables: done
Writing inode tables: done
Creating journal (65536 blocks): done
Writing superblocks and filesystem accounting information: done

root@tirpitz:~> blkid /dev/sda2
/dev/sda2: PARTUUID="e6077cf7-02"
root@tirpitz:~> mount /dev/sda2 /srv
mount: wrong fs type, bad option, bad superblock on /dev/sda2,
missing codepage or helper program, or other error

In some cases useful info is found in syslog - try
dmesg | tail or so.
root@tirpitz:~>

Does anyone have a clue what could be the issue with glibc?

glibc commit 071af4769fcdfe2cd349157b01f27c9571478ace looks like a
likely culprit. It was between 2.23 and 2.24 and is related to this
musl commit which may be easier to understand:

https://git.musl-libc.org/cgit/musl/commit/?id=7cc3a28eed7e3b72a1ce4568fb55154bb0540726

I suspect glibc did the opposite change that was recently made in musl
and went from working->broken rather than broken->working.

Debugging this is complicated by the fact that qemu-sh4 is _wrong_,
i.e. mismatches the real kernel in the opposite direction, so if
someone tested on qemu they would find it working (or if they fixed it
on real hardware, you would find it broken on qemu). Someone should
file a qemu bug for this too but I haven't gotten around to it yet.

Rich

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi!

We have recently run into a rather odd regression with glibc on Debian/sh4. Starting with glibc 2.24, file offsets are apparently wrong which results
into all weird side effects.

One of these effects is that many applications error out with "File too large", for example firebird3.0 during build:

sh -x -c "lockfile -1 /<<PKGBUILDDIR>>/gen/Release/firebird/bin/build-db.lock && /<<PKGBUILDDIR>>/gen/Release/firebird/bin/isql -q -i
/<<PKGBUILDDIR>>/src/dbs/metadata.sql; res=\$?; rm -f /<<PKGBUILDDIR>>/gen/Release/firebird/bin/build-db.lock; exit \$res"
+ lockfile -1 /<<PKGBUILDDIR>>/gen/Release/firebird/bin/build-db.lock
ERROR: ld.so: object 'libeatmydata.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
+ /<<PKGBUILDDIR>>/gen/Release/firebird/bin/isql -q -i /<<PKGBUILDDIR>>/src/dbs/metadata.sql
can't format message 17:0 -- message file /usr/lib/sh4-linux-gnu/firebird/3.0/firebird.msg not found
I/O error during "write" operation for file "/<<PKGBUILDDIR>>/gen/metadata.fdb" -Error while trying to write to file
-File too large
can't format message 17:120 -- message file /usr/lib/sh4-linux-gnu/firebird/3.0/firebird.msg not found
+ res=1
+ rm -f /<<PKGBUILDDIR>>/gen/Release/firebird/bin/build-db.lock
+ exit 1

Or the systemd-journal:

[ 13.088000] systemd-journald[524]: Failed to create new system journal: File too large

Another side effect is that mkfs does not work properly anymore since it writes the superblock
at an incorrect offset. With a properly working glibc, the superblock is written at an offset
of 0x400 while with the broken glibc (2.24), the superblock starts at 0x000. As a result,
it's not possible to create a usable filesystem on affected machines:

root@tirpitz:~> mkfs.ext4 /dev/sda2
mke2fs 1.43.3 (04-Sep-2016)
Suggestion: Use Linux kernel >= 3.18 for improved stability of the metadata and journal checksum features.
Creating filesystem with 16252928 4k blocks and 4063232 inodes
Filesystem UUID: 7d9bf1ee-a913-45d0-895b-4032020c1f3b
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424

Allocating group tables: done
Writing inode tables: done
Creating journal (65536 blocks): done
Writing superblocks and filesystem accounting information: done

root@tirpitz:~> blkid /dev/sda2
/dev/sda2: PARTUUID="e6077cf7-02"
root@tirpitz:~> mount /dev/sda2 /srv
mount: wrong fs type, bad option, bad superblock on /dev/sda2,
missing codepage or helper program, or other error

In some cases useful info is found in syslog - try
dmesg | tail or so.
root@tirpitz:~>

Does anyone have a clue what could be the issue with glibc?

Thanks,
Adrian

--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer - glaubitz@debian.org
`. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

This is a multi-part message in MIME format.
On 10/21/2016 08:36 PM, Rich Felker wrote:

glibc commit 071af4769fcdfe2cd349157b01f27c9571478ace looks like a
likely culprit. It was between 2.23 and 2.24 and is related to this
musl commit which may be easier to understand:

Indeed that looks very suspicious, thanks for pointing this out.

James Clarke (CC'ed) has suggested the attached patch on IRC which I'm testing right now.

Adrian

--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer - glaubitz@debian.org
`. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

diff --git a/sysdeps/unix/sysv/linux/sh/pread.c b/sysdeps/unix/sysv/linux/sh/pread.c
index d3f99f3..be8efb1 100644
--- a/sysdeps/unix/sysv/linux/sh/pread.c
+++ b/sysdeps/unix/sysv/linux/sh/pread.c
@@ -19,5 +19,5 @@
/* SH4 ABI does not really require argument alignment for 64-bits, but
the kernel interface for pread adds a dummy long argument before the
offset. */
-#define __ALIGNMENT_ARG
+#define __ALIGNMENT_ARG 0,
#include <sysdeps/unix/sysv/linux/pread.c>
diff --git a/sysdeps/unix/sysv/linux/sh/pread64.c b/sysdeps/unix/sysv/linux/sh/pread64.c
index b2e8a25..5d1eaac 100644
--- a/sysdeps/unix/sysv/linux/sh/pread64.c
+++ b/sysdeps/unix/sysv/linux/sh/pread64.c
@@ -19,5 +19,5 @@
/* SH4 ABI does not really require argument alignment for 64-bits, but
the kernel interface for pread adds a dummy long argument before the
offset. */
-#define __ALIGNMENT_ARG
+#define __ALIGNMENT_ARG 0,
#include <sysdeps/unix/sysv/linux/pread64.c>
diff --git a/sysdeps/unix/sysv/linux/sh/pwrite.c b/sysdeps/unix/sysv/linux/sh/pwrite.c
index 391ed5e..329c64f 100644
--- a/sysdeps/unix/sysv/linux/sh/pwrite.c
+

On 10/22/2016 03:36 AM, Rich Felker wrote:

Debugging this is complicated by the fact that qemu-sh4 is _wrong_,
i.e. mismatches the real kernel in the opposite direction, so if
someone tested on qemu they would find it working (or if they fixed it
on real hardware, you would find it broken on qemu). Someone should
file a qemu bug for this too but I haven't gotten around to it yet.

Just a quick heads-up:

The bug was indeed introduced by this commit and it was therefore easy
to fix for the original author. After verifying his suggested patch fixed
the issue (I tested his patch on my SH7785LCR), he's committed the patch
to glibc upstream [1,2].

Adrian

[1] https://sourceware.org/git/?p=glibc.git;a=commit;h=2152cf7edadb7673ea1e216de6fdfa658ec4ac1f
[2] https://sourceware.org/git/?p=glibc.git;a=commit;h=ccebcff824e76f2d3905185b8ae9ff1b51607791

--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer - glaubitz@debian.org
`. `' Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
`- GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)