• Bug#1103296: systemd: Caught from our own process

    From Peter Palfrader@21:1/5 to All on Wed Apr 16 11:00:01 2025
    Package: systemd
    Version: 252.36-1~deb12u1
    Severity: grave

    Since upgrading from 252.33-1~deb12u1, we have systemd fall over with
    | systemd[1]: Caught <ABRT> from our own process.
    | systemd[1]: Caught <ABRT>, dumped core as pid 797507.
    | systemd[1]: Freezing execution.

    rendering the system near unusable as any following systemd interactions
    time out:

    | dbus-daemon[1135]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)

    This happens on two of our systems.

    A contributing factor might be that we have a socket activated service
    that runs only for a tiny amount of time but gets triggered at a rate of
    about 10/sec:

    | [Socket]
    | ListenStream=13
    | Accept=true

    | [Service]
    | DynamicUser=true
    | ProtectHome=true
    | PrivateUsers=true
    | StandardOutput=socket
    | ExecStart=-/usr/bin/date

    Here's the bt:

    GNU gdb (Debian 13.1-3) 13.1
    Copyright (C) 2023 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    Type "show copying" and "show warranty" for details.
    This GDB was configured as "x86_64-linux-gnu".
    Type "show configuration" for configuration details.
    For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>.
    Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

    For help, type "help".
    Type "apropos word" to search for commands related to "word"...
    Reading symbols from /bin/systemd...
    Reading symbols from /usr/lib/debug/.build-id/28/c15cf513707db4d90a5476fe8be9b0e3228ebd.debug...
    [New LWP 2522914]
    [New LWP 1]
    [New LWP 2522913]
    [Thread debugging using libthread_db enabled]
    Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
    Core was generated by `/sbin/init'.
    Program terminated with signal SIGABRT, Aborted.
    #0 0x00007f599de5b267 in __GI_kill () at ../sysdeps/unix/syscall-template.S:120
    [Current thread is 1 (LWP 2522914)]
    (gdb) #0 0x00007f599de5b267 in __GI_kill () at ../sysdeps/unix/syscall-template.S:120
    #1 0x000055f59aab78bf in crash (sig=6, siginfo=0x7ffc06b8f7b0, context=<optimized out>) at ../src/core/crash-handler.c:83
    #2 <signal handler called>
    #3 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
    #4 0x00007f599dea9f4f in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
    #5 0x00007f599de5afb2 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
    #6 0x00007f599de45472 in __GI_abort () at ./stdlib/abort.c:79
    #7 0x00007f599de9e42f in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f599dfb8459 "%s\n") at ../sysdeps/posix/libc_fatal.c:156
    #8 0x00007f599deb386a in malloc_printerr (str=str@entry=0x7f599dfbb5f0 "malloc(): unsorted double linked list corrupted") at ./malloc/malloc.c:5660
    #9 0x00007f599deb6b4c in _int_malloc (av=av@entry=0x7f599dff1c60 <main_arena>, bytes=bytes@entry=800) at ./malloc/malloc.c:4006
    #10 0x00007f599deb87a2 in __libc_calloc (n=n@entry=1, elem_size=elem_size@entry=800) at ./malloc/malloc.c:3674
    #11 0x00007f599e1ceb87 in sd_bus_message_new (bus=bus@entry=0x55f5ab7c7880, m=m@entry=0x7ffc06b906f0, type=type@entry=4 '\004') at ../src/libsystemd/sd-bus/bus-message.c:467
    #12 0x00007f599e1cee2e in sd_bus_message_new_signal (bus=0x55f5ab7c7880, m=0x7ffc06b90798,
    path=0x55f5ab64c170 "/org/freedesktop/systemd1/unit/cnva_2ddaytime_40413071_2d172_2e17_2e0_2e0_3a13_2d172_2e17_2e0_2e86_3a61068_2eservice",
    interface=0x7f599e254fd8 "org.freedesktop.DBus.Properties", member=0x7f599e259f90 "PropertiesChanged") at ../src/libsystemd/sd-bus/bus-message.c:504
    #13 0x00007f599e1dbaf4 in emit_properties_changed_on_interface (bus=bus@entry=0x55f5ab7c7880, prefix=prefix@entry=0x55f5ab652890 "/org/freedesktop/systemd1/unit",
    path=path@entry=0x55f5ab64c170 "/org/freedesktop/systemd1/unit/cnva_2ddaytime_40413071_2d172_2e17_2e0_2e0_3a13_2d172_2e17_2e0_2e86_3a61068_2eservice",
    interface=interface@entry=0x7f599e258f78 "org.freedesktop.systemd1.Service", require_fallback=require_fallback@entry=true, found_interface=found_interface@entry=0x7ffc06b90867,
    names=0x0) at ../src/libsystemd/sd-bus/bus-objects.c:2088
    #14 0x00007f599e1dc4be in sd_bus_emit_properties_changed_strv (bus=bus@entry=0x55f5ab7c7880,
    path=path@entry=0x55f5ab64c170 "/org/freedesktop/systemd1/unit/cnva_2ddaytime_40413071_2d172_2e17_2e0_2e0_3a13_2d172_2e17_2e0_2e86_3a61068_2eservice",
    interface=0x7f599e258f78 "org.freedesktop.systemd1.Service", names=names@entry=0x0) at ../src/libsystemd/sd-bus/bus-objects.c:2313
    #15 0x00007f599e473960 in send_changed_signal (bus=0x55f5ab7c7880, userdata=0x55f5ab9a9e90) at ../src/core/dbus-unit.c:1612
    #16 0x00007f599e478cb7 in bus_foreach_bus (m=0x55f5ab62c290, subscribed2=0x0, send_message=0x7f599e473920 <send_changed_signal>, userdata=0x55f5ab9a9e90) at ../src/core/dbus.c:1123
    #17 0x00007f599e473baa in bus_unit_send_change_signal (u=u@entry=0x55f5ab9a9e90) at ../src/core/dbus-unit.c:1637
    #18 0x00007f599e473d30 in bus_unit_send_removed_signal (u=0x55f5ab9a9e90) at ../src/core/dbus-unit.c:1727
    #19 0x00007f599e501ad0 in unit_free (u=0x55f5ab9a9e90) at ../src/core/unit.c:682
    #20 0x00007f599e4b6cb5 in manager_dispatch_cleanup_queue (m=0x55f5ab62c290) at ../src/core/manager.c:1163
    #21 0x00007f599e4bef7e in manager_loop (m=m@entry=0x55f5ab62c290) at ../src/core/manager.c:3071
    #22 0x000055f59aab1092 in invoke_main_loop (ret_error_message=0x7ffc06b90ad8, ret_switch_root_init=<synthetic pointer>, ret_switch_root_dir=<synthetic pointer>, ret_fds=0x7ffc06b90ac8,
    ret_shutdown_verb=<synthetic pointer>, ret_retval=<synthetic pointer>, saved_rlimit_memlock=0x7ffc06b90b00, saved_rlimit_nofile=0x7ffc06b90b10, m=0x55f5ab62c290)
    at ../src/core/main.c:1909
    #23 main (argc=1, argv=<optimized out>) at ../src/core/main.c:2998
    (gdb)

    thread apply all bt full attached.
    --
    | .''`. ** Debian **
    Peter Palfrader | : :' : The universal
    https://www.palfrader.org/ | `. `' Operating System
    | `- https://www.debian.org/

    GNU gdb (Debian 13.1-3) 13.1
    Copyright (C) 2023 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    Type "show copying" and "show warranty" for details.
    This GDB was configured as "x86_64-linux-gnu".
    Type "show configuration" for configuration details.
    For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>.
    Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

    For help, type "help".
    Type "apropos word" to search for commands related to "word"...
    Reading symbols from /bin/systemd...
    Reading symbols from /usr/lib/debug/.build-id/28/c15cf513707db4d90a5476fe8be9b0e3228ebd.debug...
    [New LWP 2522914]

    warning: Unexpected size of section `.reg-xstate/2522914' in core file.
    [New LWP 1]
    [New LWP 2522913]
    [Thread debugging using libthread_db enabled]
    Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
    Core was generated by `/sbin/init'.
    Program terminated with signal SIGABRT, Aborted.

    warning: Unexpected size of section `.reg-xstate/2522914' in core file.
    #0 0x00007f599de5b267 in __GI_kill ()
    at ../sysdeps/unix/syscall-template.S:120
    120 ../sysdeps/unix/syscall-template.S: No such file or directory.
    [Current thread is 1 (LWP 2522914)]
    (gdb)
    Thread 3 (Thread 0x7f5996ffd6c0 (LWP 2522913)):
    warning: Couldn't find general-purpose registers in core file.
    #0 <unavailable> in ?? ()
    PC unavailable, cannot determine locals.
    Backtrace stopped: not enough registers or memory available to unwind further

    Thread 2 (Thread 0x7f599da8a940 (LWP 1)):
    warning: Couldn't find general-purpose registers in core file.
    #0 <unavailable> in ?? ()
    PC unavailable, cannot determine locals.
    Backtrace stopped: not enough registers or memory available to unwind further

    Thread 1 (LWP 2522914):
    #0 0x00007f599de5b267 in __GI_kill () at ../sysdeps/unix/syscall-template.S:120
    No locals.
    #1 0x000055f59aab78bf in crash (sig=6, siginfo=0x7ffc06b8f7b0, context=<optimized out>) at ../src/core/crash-handler.c:83
    sa = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {0 <repeats 16 times>}}, sa_flags = 0, sa_restorer = 0x0}
    pid = <optimized out>
    __func__ = "crash"
    __PRETTY_FUNCTION__ = "crash"
    #2 <signal handler called>
    No locals.
    #3 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
    tid = <optimized out>
    ret = 0
    pd = <optimized out>
    old_mask = {__val = {0}}
    ret = <optimized out>
    #4 0x00007f599dea9f4f in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
    No locals.
    #5 0x00007f599de5afb2 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
    ret = <optimized out>
    #6 0x00007f599de45472 in __GI_abort () at ./stdlib/abort.c:79
    save_stage = 1
    act = {__sigaction_handler = {sa_handler = 0x20, sa_sigaction = 0x20}, sa_mask = {__val = {0, 140720421274983, 140022878207472, 3688787020977367412, 3328783758931343152, 4001536, 140022762166672, 140022762166672, 140022762166672, 0,
    140022753775616, 8387968, 140022762165952, 0, 0, 0}}, sa_flags = 1596086272, sa_restorer = 0x7f5996ffd6c0}
    #7 0x00007f599de9e42f in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f599dfb8459 "%s\n") at ../sysdeps/posix/libc_fatal.c:156
    ap = {{gp_offset = 24, fp_offset = 0, overflow_arg_area = 0x7ffc06b90610, reg_save_area = 0x7ffc06b905a0}}
    fd = <optimized out>
    list = <optimized out>
    nlist = <optimized out>
    cp = <optimized out>
    #8 0x00007f599deb386a in malloc_printerr (str=str@entry=0x7f599dfbb5f0 "malloc(): unsorted double linked list corrupted") at ./malloc/malloc.c:5660
    No locals.
    #9 0x00007f599deb6b4c in _int_malloc (av=av@entry=0x7f599dff1c60 <main_arena>, bytes=bytes@entry=800) at ./malloc/malloc.c:4006
    next = <optimized out>
    iters = <optimized out>
    nb = <optimized out>
    idx = <optimized out>
    bin = <optimized out>
    victim = <optimized out>
    size = <optimized out>
    victim_index = <optimized out>
    remainder = <optimized out>
    remainder_size = <optimized out>
    block = <optimized out>
    bit = <optimized out>
    map = <optimized out>
    fwd = <optimized out>
    bck = <optimized out>
    tcache_unsorted_count = 0
    tcache_nb = <optimized out>
    tc_idx = 49
    return_cached = <optimized out>
    __PRETTY_FUNCTION__ = "_int_malloc"
    #10 0x00007f599deb87a2 in __libc_calloc (n=n@entry=1, elem_size=elem_size@entry=800) at ./malloc/malloc.c:3674
    av = <optimized out>
    oldtop = 0x55f5abaa30f0
    sz = 800
    oldtopsize = 605968
    mem = <optimized out>
    clearsize = <optimized out>
    nclears = <optimized out>
    d = <optimized out>
    bytes = 800
    __PRETTY_FUNCTION__ = "__libc_calloc"
    p = <optimized out>
    csz = <optimized out>
    #11 0x00007f599e1ceb87 in sd_bus_message_new (bus=bus@entry=0x55f5ab7c7880, m=m@entry=0x7ffc06b906f0, type=type@entry=4 '\004') at ../src/libsystemd/sd-bus/bus-message.c:467
    t = <optimized out>
    __PRETTY_FUNCTION__ = "sd_bus_message_new"
    #12 0x00007f599e1cee2e in sd_bus_message_new_signal (bus=0x55f5ab7c7880, m=0x7ffc06b90798, path=0x55f5ab64c170 "/org/freedesktop/systemd1/unit/cnva_2ddaytime_40413071_2d172_2e17_2e0_2e0_3a13_2d172_2e17_2e0_2e86_3a61068_2eservice", interface=
    0x7f599e254fd8 "org.freedesktop.DBus.Properties", member=0x7f599e259f90 "PropertiesChanged") at ../src/libsystemd/sd-bus/bus-message.c:504
    t = <optimized out>
    r = <optimized out>
    __PRETTY_FUNCTION__ = "sd_bus_message_new_signal"
    #13 0x00007f599e1dbaf4 in emit_properties_changed_on_interface (bus=bus@entry=0x55f5ab7c7880, prefix=prefix@entry=0x55f5ab652890 "/org/freedesktop/systemd1/unit", path=path@entry=0x55f5ab64c170 "/org/freedesktop/systemd1/unit/cnva_2ddaytime_40413071_
    2d172_2e17_2e0_2e0_3a13_2d172_2e17_2e0_2e86_3a61068_2eservice", interface=interface@entry=0x7f599e258f78 "org.freedesktop.systemd1.Service", require_fallback=require_fallback@entry=true, found_interface=found_interface@entry=0x7ffc06b90867, names=0x0) at
    ../src/libsystemd/sd-bus/bus-objects.c:2088
    error = {name = 0x0, message = 0x0, _need_free = 0}
    m = 0x0
    has_invalidating = false
    has_changing = false
    key = {path = 0x0, interface = 0x0, member = 0x0, parent = 0x0, last_iteration = 0, vtable = 0x0}
    n = 0x55f5ab7d0800
    u = 0x0
    r = <optimized out>
    __PRETTY_FUNCTION__ = "emit_properties_changed_on_interface"
    #14 0x00007f599e1dc4be in sd_bus_emit_properties_changed_strv (bus=bus@entry=0x55f5ab7c7880, path=path@entry=0x55f5ab64c170 "/org/freedesktop/systemd1/unit/cnva_2ddaytime_40413071_2d172_2e17_2e0_2e0_3a13_2d172_2e17_2e0_2e86_3a61068_2eservice", interface=
    0x7f599e258f78 "org.freedesktop.systemd1.Service", names=names@entry=0x0) at ../src/libsystemd/sd-bus/bus-objects.c:2313
    _slash = <optimized out>
    prefix = 0x55f5ab652890 "/org/freedesktop/systemd1/unit"
    found_interface = false
    pl = <optimized out>
    r = <optimized out>
    _dont_destroy_bus = <optimized out>
    __PRETTY_FUNCTION__ = "sd_bus_emit_properties_changed_strv"
    #15 0x00007f599e473960 in send_changed_signal (bus=0x55f5ab7c7880, userdata=0x55f5ab9a9e90) at ../src/core/dbus-unit.c:1612
    p = 0x55f5ab64c170 "/org/freedesktop/systemd1/unit/cnva_2ddaytime_40413071_2d172_2e17_2e0_2e0_3a13_2d172_2e17_2e0_2e86_3a61068_2eservice"
    u = 0x55f5ab9a9e90
    r = <optimized out>
    __PRETTY_FUNCTION__ = "send_changed_signal"
    #16 0x00007f599e478cb7 in bus_foreach_bus (m=0x55f5ab62c290, subscribed2=0x0, send_message=0x7f599e473920 <send_changed_signal>, userdata=0x55f5ab9a9e90) at ../src/core/dbus.c:1123
    b = 0x0
    r = <optimized out>
    ret = 0
    #17 0x00007f599e473baa in bus_unit_send_change_signal (u=u@entry=0x55f5ab9a9e90) at ../src/core/dbus-unit.c:1637
    r = <optimized out>
    __PRETTY_FUNCTION__ = "bus_unit_send_change_signal"
    __func__ = "bus_unit_send_change_signal"
    #18 0x00007f599e473d30 in bus_unit_send_removed_signal (u=0x55f5ab9a9e90) at ../src/core/dbus-unit.c:1727
    r = <optimized out>
    __PRETTY_FUNCTION__ = "bus_unit_send_removed_signal"
    __func__ = "bus_unit_send_removed_signal"
    _error = <optimized out>
    _u = <optimized out>
    _l = <optimized out>
    #19 0x00007f599e501ad0 in unit_free (u=0x55f5ab9a9e90) at ../src/core/unit.c:682
    slice = <optimized out>
    t = 0x0
    __PRETTY_FUNCTION__ = "unit_free"
    #20 0x00007f599e4b6cb5 in manager_dispatch_cleanup_queue (m=0x55f5ab62c290) at ../src/core/manager.c:1163
    u = <optimized out>
    n = 0
    __PRETTY_FUNCTION__ = "manager_dispatch_cleanup_queue"
    #21 0x00007f599e4bef7e in manager_loop (m=m@entry=0x55f5ab62c290) at ../src/core/manager.c:3071
    rl = {interval = 1000000, burst = 50000, num = 268, begin = 777735039283}
    r = <optimized out>
    __PRETTY_FUNCTION__ = "manager_loop"
    __func__ = "manager_loop"
    #22 0x000055f59aab1092 in invoke_main_loop (ret_error_message=0x7ffc06b90ad8, ret_switch_root_init=<synthetic pointer>, ret_switch_root_dir=<synthetic pointer>, ret_fds=0x7ffc06b90ac8, ret_shutdown_verb=<synthetic pointer>, ret_retval=<synthetic pointer>,
    saved_rlimit_memlock=0x7ffc06b90b00, saved_rlimit_nofile=0x7ffc06b90b10, m=0x55f5ab62c290) at ../src/core/main.c:1909
    objective = <optimized out>
    table = <optimized out>
    _level = <optimized out>
    _e = <optimized out>
    r = <optimized out>
    r = <optimized out>
    __PRETTY_FUNCTION__ = <optimized out>
    __func__ = <optimized out>
    objective = <optimized out>
    _error = <optimized out>
    _level = <optimized out>
    _e = <optimized out>
    saved_log_target = <optimized out>
    saved_log_level = <optimized out>
    _level = <optimized out>
    _e = <optimized out>
    _level = <optimized out>
    _e = <optimized out>
    _level = <optimized out>
    _e = <optimized out>
    _pptr_ = <optimized out>
    _ptr_ = <optimized out>
    _pptr_ = <optimized out>
    _ptr_ = <optimized out>
    _level = <optimized out>
    _e = <optimized out>
    table = <optimized out>
    _level = <optimized out>
    _e = <optimized out>
    #23 main (argc=1, argv=<optimized out>) at ../src/core/main.c:2998
    initrd_timestamp = <optimized out>
    userspace_timestamp = {realtime = 1744010878661534, monotonic = 4383650}
    kernel_timestamp = {realtime = 1744010874277883, monotonic = 0}
    security_start_timestamp = {realtime = 1744010878668210, monotonic = 4390326}
    security_finish_timestamp = {realtime = 1744010878670670, monotonic = 4392787}
    saved_rlimit_nofile = {rlim_cur = 1024, rlim_max = 4096}
    saved_rlimit_memlock = {rlim_cur = 8388608, rlim_max = 8388608}
    skip_setup = <optimized out>
    loaded_policy = false
    queue_default_job = <optimized out>
    first_boot = false
    switch_root_dir = 0x0
    switch_root_init = 0x0
    before_startup = 4879425
    after_startup = <optimized out>
    shutdown_verb = 0x0
    error_message = 0x0
    r = <optimized out>
    retval = 1
    m = 0x55f5ab62c290
    fds = 0x0
    __PRETTY_FUNCTION__ = "main"
    systemd = "systemd"
    __func__ = "main"
    _found = <optimized out>
    __assert_in_set = <optimized out>
    (gdb)

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Luca Boccassi@21:1/5 to All on Wed Apr 16 13:00:02 2025
    Control: severity -1 normal

    On Wed, 16 Apr 2025 08:38:56 +0000 Peter Palfrader <weasel@debian.org>
    wrote:
    Package: systemd
    Version: 252.36-1~deb12u1
    Severity: grave

    Since upgrading from 252.33-1~deb12u1, we have systemd fall over with
    | systemd[1]: Caught <ABRT> from our own process.
    | systemd[1]: Caught <ABRT>, dumped core as pid 797507.
    | systemd[1]: Freezing execution.

    rendering the system near unusable as any following systemd
    interactions
    time out:

    | dbus-daemon[1135]: [system] Failed to activate service
    'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)

    This happens on two of our systems.

    Would you be able to provide the core file too?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Debian Bug Tracking System@21:1/5 to All on Wed Apr 16 13:00:01 2025
    Processing control commands:

    severity -1 normal
    Bug #1103296 [systemd] systemd: Caught <ABRT> from our own process
    Severity set to 'normal' from 'grave'

    --
    1103296: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1103296
    Debian Bug Tracking System
    Contact owner@bugs.debian.org with problems

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)